We have several Crawlers on our account and since the Crawler works as a web scraper I was curious if we should expect to see (albeit probably minor) increases in traffic on crawled pages in our Website Analytics Platform? (we use google Analytics)
In addition, if we do expect see spikes what would be the best way to target crawler page hits to filter it out?
I’m not sure about the crawler behavior yet–just getting started with Yext–but they do list the IP addresses and user agents that the crawler uses in this article. See below for an excerpt with the IP addresses.
I believe the following regular expression will match the IP addresses. You could configure that as a filter in your analytics application to exclude that traffic.
(?:54\.204\.19\.87|50\.19\.160\.200|34\.198\.218\.97|54\.221\.171\.225)