With this release, we have added a configurable rate limit for the crawler so businesses can ensure their sites are not crawled too quickly and prevent sites from being overwhelmed by excessive requests from crawlers.
Users will now be able to add a definitive rate limit value, allowing them to decide the maximum number of concurrent requests that can run on their site. The default rate limit will be 100 concurrent crawls, but users can change the rate limit to any integer up to a maximum for 15,000.
When setting up a Crawler, you will see a new Advanced Crawler Settings section. In this section, you can add your Rate Limit.
Turn on the Spring ‘22 Release: Crawler Rate Limit (early access) account feature to use this feature during the Early Access period.
To learn more about Crawlers, visit the Yext Site Crawler & Crawler Connector training module.