If you’re already a Hitchhiker, log in to access this content.
Reviewing & Scoring Search Terms| Hitchhikers Platform
What You’ll Learn
In this section, you will learn:
- How to see, review, and export search terms
- How to review search terms and add status
- How to score search term relevance for benchmarking
- How to add labels to Search Terms
The best way to analyze search trends and evaluate search quality is by looking at search terms instead of individual searches. (You can view and export individual searches by both external users in your live experience and internal users using Test Search in the Search Logs tab under Answers > Experiences > Analysis.) The Search Terms screen, which you can find right under the Search Logs tab, normalizes query strings (for things like capitalization, leading and trailing whitespace, and punctuation) into a list of unique search terms run by your users, and by default sorts the search terms by popularity.
From this screen, you can:
- Filter search terms by date, results returned, status and more (1)
- Scroll through the search terms and view key data points about each search term like Sessions, Searches with Click and CTR (Click Through Rate) (7)
- Set Search Term status for an individual Search Term (8) or in bulk (3)
- Enter a workflow to review selected Search Terms (4) to set both Status and Entity Relevancy
- Export all or filtered search terms into a spreadsheet (2)
- Bulk upload labels (6)
- Run the same search in Staging or Production (10)
- Add labels to Search Terms for an individual Search Term (9) or in bulk (5)
We’ll go into more detail on each of these items below.
Filter Search Terms
You’ll see all filters currently applied in the filter bar at the top of the screen. To add a filter, you can either click on a filter pill, or click on the + icon to open the filter window.
Search Term Detail
The Search Terms screen gives you a high level overview of key data points for each search term. When you click into a search term, you can view additional details about that search term, including:
- The most recent search details, like language and location bias (for context)
- Most Recent Knowledge Graph Results for the Search Term
- Relevancy Score for Entities Returned
- The Search Term Status
- The Search Term Activity
- Additional links, like “Open Search in Production”
Search Term Status
As an Administrator, your job is to make sure that a brand’s customers get answers. This means maintaining a high level of search quality. But, it can be hard to keep track of where you need to do work or make improvements.
Search Term Status allows you to set a status and notes on a query so that you can manage your workflows efficiently. You have the option of setting a query to:
- No status (default)
- Looks Good – this means you’ve reviewed the results and feel confident that the results are answering the question
- Needs Review – this means that you’ve reviewed the results and may need to do some work to improve the results
You can accompany the status with notes to keep track of why you set the status that way, or even with a list of to-dos. Or, you can have other people set the status and use the notes field to communicate to you why they think the status “looks good” or “needs review”. Either way, these two features are here to make your workflows and operations easier so use them as you see fit.
In general, we recommend:
Focus on Most Popular Queries: In general, we recommend that you make sure that the search terms that make up 60-80% of your search volume have a status set. There will always be a significant long-tail of single-time search terms – while it’s important to deliver quality answers for all your consumer’s questions, your primary focus should be on addressing the most popular questions first.
Bulk Setting Needs Review: You can filter to
Results Returned= No Knowledge Graph Results, select all results and bulk set the query status to Needs Review. This will help you to quickly identify those queries that don’t have structured results as a starting point for your review.
- Use the Workflow to review search terms consecutively: You can bulk select search terms, e.g., all with status set to Needs Review or no status set yet and click on “More Actions > View Search Terms”. This will enter you into a Review flow and when you’re finished with a query you can click the Save & Next button to review the next. This should save you time!
Scoring Entity Relevancy for Benchmarking
While the Status is there for your workflow efficiency, Scoring Entity Relevancy is your most powerful tool for improving your search results in the long term.
Instead of evaluating whether a query is “good” or “bad”, Entity Relevancy allows you to be even more granular by evaluating whether a result is the “answer”, “relevant” or “not relevant” to the customer’s question (Search Term). This information can then be used in two incredibly important ways:
Evaluating the Impact of Search Configuration Changes: As you make changes to your search configuration, it’s important to have a training set of data to understand whether those changes are going to improve or degrade your search results. For example, imagine you make a change to your configuration that removes entities from a search term that you’ve previously marked as irrelevant – that’s an improvement! However if that same change removes entities that you previously marked as relevant, then you’ll want to consider whether you should push that change. We call this Version Comparison and you’ll learn more about it in the Version Comparison unit in this module.
Overall Impact on Algorithm Changes: On a macro scale, your input on whether results are relevant will be the best way to help to inform Yext’s Product Managers, Engineers and Data Scientists about the algorithm, how it’s performing for your businesses’ needs and how it can be improved overall. By scoring the relevancy of Entities, you are able to tell us at Yext what you think of as good search for your business and how we can improve search for you.
How to Score Entity Relevance
For each search term, we’ll surface each of the entities that are returned today so that you can judge whether those results are the Answer, Relevant or Not Relevant to the Search Term.
Relevant means that the entity should be returned in the search results and is relevant to what the user is looking for.
Not Relevant means that ideally the entity should not be returned in the search results because it is not related to the search term.
Answer can only be used for one entity and indicates that you think that entity should be the Direct Answer for the user’s query.
You can click to set the relevancy, or you can use the bulk selection and buttons to do multiple at once. For example:
Search Term Detail Workflow
While Status and Entity Relevancy are two separate features, you can set them both as part of a single workflow and we strongly encourage you to be diligent about both.
As mentioned earlier, you can enter a workflow by bulk selecting search terms on the Search Terms screen and selecting More Actions > View Search Terms. When you do this, you can set the Entity Relevancy and set the status of the query or add notes in one session. Then, you can click Save & Next to go to the next query. This should make your process and workflow much faster!
Keep in mind that you can have search terms that “Need Review” but most entities are scored as Relevant – or the inverse, you can have search terms that are “Looks Good” even though there are entities that are Not Relevant returned. These are independent ways of evaluating queries, so make sure to think about them separately.
Search Term Labels
As consumers continue to search for things in new and unique ways, we’re looking for ways to categorize questions consumers are asking to understand and improve search at scale. Search Term Labels go hand in hand with Search Term Clusters, which automatically groups similar searches, because it gives Hitchhikers more control over how search terms are grouped.
Search Term Labels can be added individually or in bulk. To add individually, click + Add a Label under the Labels column, enter relevant labels, and hit Save.
You can also label Search Terms in bulk by selecting multiple Search Terms and clicking Manage Labels under More Actions. From there you can add labels, remove labels, or overwrite all existing labels with newly specified labels.
Once you add Search Term Labels, you can use them for the following purposes: