My client is pursuing using GCSE on Answers, but is maintaining COVEO (another site search provider) on their other sites that will index the same domains.
They reached out to me with the below questions - is anyone able to assist?
How often are these sites being indexed?
Is there a way we can tell Google to only crawl the site at certain times? Don’t want to overload the server
Hi Jessie,
Google Custom Search Engines don’t explicitly index the domains that are provided to be included. Rather, those domain patterns are used to decide what to include in the search results from Google’s existing index.
This is why it’s important that a client maintain their robots.txt file and sitemap so that Google itself is indexing the right content - this then will impact what is available to be featured in the clients GCSE.
In short, there should be no impact on their servers as we’re leveraging what has already been indexed by Google bots.
Hope that helps!
Thanks so much Amani! This is really helpful.