What happens to entities if the crawled/connected page is taken down?

Hi! We are crawling articles from a client’s pages and then using the data connector to pull in those pages as Help Article entities. This data is being used for an Extractive QA experience. The concern is if an article is taken down by the client, what happens the next time the crawler/connector run? Would the entity be deleted?

The goal is that if the information changes the entity is updated and if the page is taken down the entity is too. This would ensure that we do not have QA answers that link out to pages that don’t exist.

Hi Jordan,

Currently, a Connector cannot delete entities. So in your case, you would have to manually remove any old Help Article entities from the account. For the summer release however, we’re introducing a new feature called Comprehensive Deletion Mode. If this mode is enabled, any entities that were created in a previous run, but are not present in the current run will be deleted. This new feature should solve the issue you raised, but until it’s released, the only way to solve this problem is via manual deletion.

Hope this helps!