Using Page Based Pagination on Pull API - Workaround

Content
1

Hello HH community,

Here is a rough workaround on the Page Based Pagination when there is no Page ID on the JSON response body.

Situation:
I had an instance recently where I was trying to Pull Event data from JSON ( e.g: https://brand.com/json?itemsPerPage=100&pageID=1) using the new Pagination functionality on the connector.

They had the following parameter on the URL:

  • itemsPerPage which defines how many events are shown
  • Page ID: Page number

Issue:
Fetching all items in one go was not possible as there are over 10,000 items and the connector timed out after 2500 in one go (without pagination). I tried playing with the parameters above as Page Key and limit key but the items pulled were only the amount specified on the itemsperpage parameter on the request URL.
In most cases, in order to use the page based pagination functionality, the Page ID needs to be on the Response Body, which in my case I did not have and could not be added.

Resolution:
As the pageID parameter was just an integer that increments each call, we got rid of the pagaination related parameters (itemsperpage and page id) from the Request URL and set the pagination up as follows:

image
image768Ă—808 38.5 KB

This allowed the Pull API to work as we are telling the system to adjust the base URL provided by putting a query parameter of itemsperpage=100 and PageID=* (that starts at 1) for every call

  • meaning with this setup we do not need to look for anything in the JSON response.

To note, the API call in Preview may fail ( this may be because the preview call seems to not include params from the pagination control section, so it is making a call for a larger set of events and taking too long)

2 Likes
2

Please note that the “Limit Key” field is required even if the source API doesn’t need or require that field. Otherwise, the Connector will just pull a single “page” and stop immediately.

1 Like