Does the crawler work on pages that paginate?

Search
1

Hi Team,

Does our crawler work on pages that paginate? The information I’m trying to crawl loads new content when you click the page number you want to be directed to; however, on the newly loaded page, the URL pattern does not change. The URL ends as a .html#paginationnub.

Currently, the crawler isn’t scraping the additional pages. Is there a work around?

Thanks!

2

Hey Marion,

The crawler can spider to pages that paginate as long as the URL updates when you paginate. For example, if page one and page two have URLs that contain #1 and #2 in the URL, we could spider to each page and find additional links. If pagination is not done this way, the crawler won’t be able to spider properly.

Thanks,
Calvin