Pull from API | Yext Hitchhikers Platform

Overview

Any data that can be retrieved via an API (formatted as JSON) can be extracted and loaded into a connector. This is referred to as the Pull from API connector source.

Building a connector using an API as the data source requires familiarity with the source’s API documentation. This is a similar process to using a Native Source connector , but instead, the connection will be configured manually.

Request Details

Method

Designate whether the request should be a GET or POST request. POST requests are generally associated with GraphQL APIs.

Request URL

Provide the URL path to the API. It is not necessary to specify any query parameters, pagination settings, or authentication settings within the URL. If they are specified in their designated sections, the system will correctly construct the final API request URL.

Authentication Method

Select the method of authentication that the API supports from the following options:

API Key: The API key will be added as a query parameter or header parameter, based on what’s specified.
Bearer Token: The provided token will be added as an Authorization header, prepended with “Bearer.”
Basic Authentication: Provide a username and password, which will then be Base 64 encoded by our system and added as an authorization header, prepended with “Bearer.”
OAuth: If OAuth is the method of authentication, you’ll need to specify a linked account. The Linked Account system manages all tokens and refresh tokens. If the provider for your API is not a built-in provider, follow this guide to create a custom OAuth provider and link your account.

Note

In some cases, the documentation for your chosen API source may not outright specify OAuth as the method of authentication. Instead, the documentation may walk through the process of obtaining a token with a separate API call, and then using that token in the API request. This process is essentially the same as OAuth. Generating and hardcoding a bearer token is not recommended whenever OAuth is supported. For the purposes of security, refresh tokens, and ease-of-use, this terminology should signify usage of OAuth.

Query Parameters

Specify any query parameters that should be appended to the final API request. Do not include authorization query parameters or pagination query parameters, as those should be provided in their respective sections.

As an example, if the following key value pair is provided as a query parameter:

Key: v
Value: 20240101

And if the request URL is https://www.myapi.com/test, the system will construct the following request: https://www.myapi.com/test?v=20230101

Headers

Specify any key-value pairs that should be added as additional headers to the API request. By default, the system adds:

Content Type: application/json
User Agent: YextConnector

Max Requests (Optional)

Specify a rate limit for the connector to adhere to. Provide the maximum number of requests the system should make to the API in a given unit of time, to ensure the rate limit is not exceeded.

Pagination

The Pull from API source accepts four pagination options:

Cursor
Link Header
Offset
Page-Based

Refer to the API documentation of your chosen API source to determine which method(s) may be supported.

For any of the methods, if Max Pages is provided, the connector will respect the provided value and not paginate beyond the maximum number of pages specified.

Cursor

APIs that utilize cursor pagination (such as the Yext APIs) will return a token or URL as part of the response body or headers, which can then be referenced as a query parameter in the next API call to fetch the next page of results.

In order for the connector to correctly paginate, it needs to know:

Cursor type: Is the cursor a token (i.e., a random string of characters), or is a full URL or relative URL returned?
Location: Is the cursor in the response body or in the header?
Cursor property: What property contains the value for the cursor?
- For example, if the cursor is in the response body, the path to property with URL may be response.nextPageUrl for an API response like the following:
```
{ 
response: {
nextPageUrl: "https://myapi.com?nextpagetoken123"
      }
}
```
  If the cursor type is token, provide the page key that should be passed as a query parameter in each API request to contain the value for the token provided in the previous page’s response. For example, the first page of the API response might look like this:
```
{ 
response: {
nextPageToken: hKLFJoin89041JKLg8hs9fjkdlsaH789HJ
      }
}
```
  To paginate, the next API request may need to look like this: https://myapi.com?pageToken=hKLFJoin89041JKLg8hs9fjkdlsaH789HJ

In this case, the page key would be pageToken, as this query param contains the value of the token provided in the previous request, for the next page.

Link Header

APIs that use link header pagination have a header property link that contains direct links to other pages in the API response, such as to the next page, last page, previous page, and/or first page, as specified by the value in the relationship (rel) label.

For example, the header might look like the following:

Link: <https://myapi.com?page=15>; rel="next",
  <https://myapi.com?page=100>; rel="last",
  <https://myapi.com?page=1>; rel="first",
  <https://myapi.com?page=13>; rel="prev"

The connector can paginate by getting access to the URL for the next page, by specifying the following two settings:

Link type: Specify whether the link that is provided is the full URL to the next page, or just the relative URL (in the example above, the full URL is provided).
Relationship label: Specify the value for rel that contains the link to the next page (in the example above, this is next).

Offset

Offset pagination relies on passing an offset parameter, to specify the offset of the item to return from the first item in the list (or, the number of items to skip when fetching the next set of results). A limit parameter is required as well, to signify the maximum number of items to return in the response.

For example, if each page should contain 10 items, and you want to paginate to retrieve items 11-20, the resulting API request may look like this: https://www.myapi.com?limit=10&offset=10

In order for the connector to correctly paginate, it needs to know:

What is the offset key? (In the above example, this is offset)
What is the value for the first item (what is the offset start value)?
- This is typically 0 or 1, but refer to the API documentation to confirm.
What is the limit key to contain the maximum entries per page? (In the above example, this is limit)
How many entries should be returned per page (what is the limit value)?
- In the above example, the value is 10. Refer to your API documentation to learn what’s supported for your API.
What should tell the connector that the final page or final item has been reached?

There are two ways to signal to the connector that the final page has been reached:

Total pages key
- If in the header: What is the key of that header containing the value? (e.g., API header contains X-TOTAL-PAGES: 10)
- If in the response body: What is the JMESpath to the node containing the value? (e.g., for the following API response body, the path is response.totalPages)
Total items key
- If in the header: What is the key of that header containing the value? (e.g., API header contains X-TOTAL-ITEMS: 10)
- If in the response body: What is the JMESpath to the node containing the value? (e.g., for the following API response body, the path is response.totalItems)

Example response:

{ 
	response: {
		items: {
[
					item1data: {}
					...
				]
			},
		totalPages: 10
		totalItems: 100
          }
}

Page-Based

Page-based pagination relies on passing a query parameter for the page of the API response to return.

For example, the resulting API request may look like: https://www.myapi.com?pageNumber=3

In order for the connector to correctly paginate, it needs to know:

What is the format of the query param (what is the page key)?
- In the above example, this is pageNumber.
What is the value of the query param for the first page (what is the start value)?
- This is typically 0 or 1, but refer to the API documentation to confirm.
What should tell the connector that the final page has been reached?

There are two ways to signal to the connector that the final page has been reached:

Total pages key
- If in the header: What is the key of that header containing the value? (e.g., API header contains X-TOTAL-PAGES: 10)
- If in the response body: What is the JMESpath to the node containing the value? (e.g., for the following API response body, the path is response.totalPages)
- Example response body:
```
{ 
response: {
items: {
        [
            item1data: {}
            ...
        ]
       },
totalPages: 10
      }
}
```
A key that contains the total number of entries returned per page (whereby if the connector reaches a page with fewer items, the final page is detected)
- Limit key: The JMESpath to the key in the response body containing the value for the maximum number of entries returned in a single page
- Limit value: (optional) provide a value for the limit key that is different from that of the default of your API source, which will be provided as a query parameter in each API call. For example, to request only 10 items per page, each API request may contain the following param: https://www.myapi.com?per_page=10
- Example response body (where the limit key is per_page and the limit value is 10 items per page)
```
{ 
response: {
items: {
        [
            item1data: {}
            ...
        ]
       },
per_page: 10
      }
}
```

Limitations

The following pagination methods are not supported for a connector using the pull from API source:

LQS pagination
Pagination that requires the pagination settings to be passed in the response body (this is very likely for any GraphQL use cases)

You will need to use the function source to properly connect to these APIs and paginate.

If your API uses OAuth, but the grant type is not “Authorization Code” or “Client Credentials,” you will also need to use the function connector source.

Back

< Function Source

Adobe Commerce >

Feedback

<% elem.innerText %>