How does the "List Page" functionality work in a Crawler Connector?

1

I’m trying to use the List Page functionality in a Crawler Connector but I’m not sure what to put in the “Specify Entity Container” field. I’m using CSS. I’m putting in the selector that contains the entities but none of the field mappings are coming through when I use CSS selectors on those. Is it a problem with the CSS selector used in the first step? How can I tell what the right value to use is? How does the selected value affect the subsequent field mapping step, if at all?

2

Hi Matt,

Before identifying specific fields that you would like to extract from your pages, you must first tell the Connectors system if a single page contains the data for one entity or for many entities.

  • If each page contains one entity, you can select “Detail Page” and move on to the “Specify Selectors” step to specify the CSS or XPath selectors for the fields that you would like to extract.

  • If each page contains a list of entities, you have to select “List Page” and provide a CSS or XPath selector that points to the outer container for each entity on the page (the “Entity Container” or “Base Selector”). For example, if we wanted to extract each FAQ on this Yext FAQ page into a separate entity, the Entity Container CSS selector could be “.faq” because the information for each FAQ is contained within the HTML element with the “faq” CSS class.

Screen Shot 2021-07-23 at 11.53.50 AM
Screen Shot 2021-07-23 at 11.53.50 AM1098×798 134 KB

Each object extracted from your page based on that Entity Container selector will be treated as an entity, or a row in your preview table. Once you provide your Entity Container, you can specify the selectors for the actual fields you want to extract – these field selectors should be specified relative to the Entity Container selector. For example, on that Yext FAQ page, you could specify the CSS Selector “.questionText” to extract the question text and “.answerText” to extract the answer text and the Connectors system will know to look for these elements within each instance of “.faq” (your Entity Container).

Hope that helps!

Thanks,
Jamie

3

Thank you Jamie, this is perfect! You really know your stuff :slight_smile: