Hi team! I am working with a client using the Crawler where we are noticing multiple urls are getting pulled in for a single article because the urls have appended corresponding identifiers which doesn’t affect the content. Therefore, we have many duplicates I want to remove.
My thinking is to set a parameter which, moving forward, would not crawl a URL if there is content after a particular /* - how exactly would I do that? Essentially, I want to exclude all URLs with the identifiers appended at the end.
Thanks in advance for the help!