Hi,
I am trying to set up connectors for a client but we cannot get past the crawler. Indeed, the crawler is blocked due to whitelisting issues. However, the client cannot whitelist our user agents because, to prevent DoS (Denial of Service) attacks, i.e. where an attacker coordinates lots of requests in order to overload and bring down our platforms, they have a configuration on the firewall that tells it to block unexpected requests. In their case, it’s a rule called: “Tools and frameworks for web crawling and attacks”.
This rule is managed by Microsoft, i.e. they update the criteria used to determine if a request is from a customer, a bot from Google or Bing, or an attacker. It’s not possible to add exclusions to that specific rule, so they can’t exclude Yext by IP address.
The ideal scenario here is for Yext to modify the user agent that we are sending.
Is it possible for you to modify the user agent we are sending?
Thanks!
Fede