You can configure the Crawler to ignore/exclude the URLs you do not want to crawl. The platform provides better control and options to achieve the same while add/edit any Crawler.
For help, you can find the detail infromation for the same on Create a Crawler | Hitchhikers
Hope it helps you to achieve the solution.