Comprehensive Run Mode | Yext Hitchhikers Platform
Overview
This reference details the behavior and nuances of connectors that are run in Comprehensive Mode. For more information on run modes, see the Run Modes reference.
Comprehensive Data Set
If a connector is run in Comprehensive Mode, the entities present in the connector run will be considered as a comprehensive data set, meaning:
- New entities present in the run will be created.
- Existing entities will be updated (or remain unchanged), based on the data present for those entities in the current run.
- Any entities that were created in a previous run of the same connector, but are not present in the current run, will be deleted.
Configuration Options
You can choose to define the comprehensive data set and whether to apply updates to missing entities in a given connector in Connector Configuration > Advanced Settings, after the Mapping stage.
By default, the comprehensive set is any entity ever created by the given connector. Optionally, you can define the comprehensive set as entities within a specified saved filter.
By default, any entities that are missing in the run from the comprehensive set (as defined above) will be deleted. Optionally, you can define a set of updates to apply to missing entities. Options for specifying updates to missing entities are detailed below.
Update Missing Entities
The flow to configure entity updates consists of three stages: Extract, Transform, and Mapping. The missing entities will be considered the “source” for this flow.
Specify Selectors
Select the fields that should be modified in the subsequent stages. As in any connector flow, not all selected fields need to be mapped — they are simply made available to include in the Transform stage.
Adding default selectors will extract the same fields that were mapped to in the primary connector flow.
To manually add selectors, specify a header for the column and select the field to extract data from.
Available fields to select include almost any enabled fields on the entity type configured for the connector. The following fields and field types are not available to extract:
- Entity Relationship type fields
- Folders
- Categories
- Linked Accounts
- ECLs
- Any field using the Rich Text v1 (Legacy Rich Text) field type
When specifying selectors, you will see a table of preview data, which will populate with the 10 most rcently updated entities of the specified entity type. Note that the preview table is populated with example entities, and is not representative of actual entities missing from the comprehensive set. This is because the actual missing entities can only be calculated at run time.
Add Transforms
Transform existing data using the library of built-in transforms, or custom function transforms.
It is not recommended to modify the Entity ID column in any way during this stage. If the Entity ID column is manipulated, the “missing set” of entities to be updated will reflect the current set of Entity IDs. This could result in unintentional updates to other entities, or unintentional entity creation, depending on the changes that were made to the Entity ID column.
Map Data
Map columns of data to fields of the entity type configured on the connector. Any columns that are not mapped will be ignored.
This behavior is the same as the Mapping stage of all connectors. For more information about mapping data, refer to the Map Fields reference.
Analyze Run Results
When running in Comprehensive mode, a new file of data will be available to download containing the list of all missing entity IDs.
Note that this file will represent the set of entity IDs considered “missing” from the comprehensive run prior to any data manipulation steps to update the missing entities. As mentioned above, manipulating the Entity ID column can modify this set.
The missing entities will be included in the counts for Created/Updated/Unchanged/Failed/Deleted entities. This means that the number of entities in each of those files may now be greater than the number of entities in the run, since it includes results for the comprehensive set.
For example: Connector A pulls in location data stored in a file, and the comprehensive set is defined as all entities within the “Open Locations” saved filter. Comprehensive mode is configured to update all entities missing from the run, but that previously existed in the “Open Locations” saved filter, to have -CLOSED appended to the Name field.
- When the file is uploaded, there are 25 entities to be updated, and five entities are considered “missing.”
- When the connector runs, the count for updated entities will be 30, to account for the 25 entities in the current run, plus the five missing entities that were updated.
Other Functionality
Run Breakers
Run breakers will only operate on the entities counted within the source data during the connector run, and the initial count of missing entities for that run. A run breaker will not consider what actually happens to the missing entities (e.g., whether they are updated) to avoid any duplicate actions on the same entity.
A common use case for run breakers is to break a run if a certain number of entities are deleted or missing from the comprehensive set. To configure this, the run breaker can be set on the comprehensive set of entities with a specified threshold for missing entities. If the number of missing entities exceeds the specified threshold, the run breaker will be triggered and the connector run will be aborted.
For more information on configuring run breakers, see the Run Breakers reference.
Entity Type Mismatch
If the entity type is changed, or if the comprehensive set is defined by a saved filter containing multiple entity types, it is possible that the entity type of “missing” entities in a run will not match the entity type configured in the source data for the run.
Any missing entities will fail to update if their entity type does not match the type specified in the source data.