Debugging Chat | Yext Hitchhikers Platform

What You’ll Learn

By the end of this unit, you will be able to:

• List and navigate to the four tools you can use to debug chat
• Identify if issues are due to gaol configuration, search configuration, or Content gaps
• Explain three prompts that are useful in debugging

Overview

This unit will focus on how to leverage tools within the platform to help you debug your Chat configuration. We’ll focus on what these tools are and what types of information you can get from them, so that you have a good starting point for debugging backend issues.

There are several useful tools at your disposal including:

• Chat Preview
• Response Logs
• Response Log Details
• Intent Insights

Each of these tools helps you understand the logic that Chat is using in order to answer a user. If the responses (or lack thereof) to users aren’t relevant then there are three key factors that could be affecting your bot:

• Goal configuration issues (example phrases and instruction setup)
• Search configuration issues
• Data gaps (in Yext Content or other external data sources)

Using Chat Preview to Debug

Chat Preview is the best place to start when debugging your bot. If you’re spotting some strange results within a conversation, or you are just making changes to your configuration, you should use Chat Preview.

If you are actively making changes to your bot configuration, make sure that you are saving the configuration and refreshing the preview. This ensures that your changes are being reflected properly in the Chat Preview.

Once you input queries into the preview, you will then be able to access those results and see the logic the bot is using to give responses in the response logs and response log details, which you will learn about next!

Using Response Logs to Debug

The response logs can show you at a high-level how well your bot is responding to recent user queries. The table on the Response Logs screen shows you each message submitted along with the bot’s response and the goal predicted by the bot.

If the correct bot goal was triggered and the bot’s response helps answer the user’s message accurately then great! There is nothing left you need to do.

However if either the bot’s message or predicted goal appear incorrect then it could tell you a couple things:

1. An incorrectly predicted goal or no predicted goal could indicate that the goal settings need to be adjusted (e.g., example phrases need to be modified or added) or that a new goal needs to be created.
2. An incorrect bot message but correct goal could indicate an issue with content (e.g., an entity needs to be added to Content) or instruction setup (e.g., properties are missing or incorrectly filled in).

Once you determine that a user message isn’t being responded to appropriately, you can also dig into the issue further with the Response Log Details screen. This screen will help determine what actions you need to take to enable your bot to respond to the user as intended.

Using Response Log Details to Debug

Goal Detection

The first section on the Response Log Details screen is to show how the bot matched the user’s query to a goal. For the predictGoal prompt, click into the response to see the logic the bot used. If the reasoning looks incorrect, adjust your goals example phrases or add a new goal.

For example, if a user asks “What is Dr. Smith’s phone number”, the bot matches that to the goal PROVIDER_QUESTIONS since it interprets the user is asking for information about a provider. However, the current goal is setup to run the user through an appointment creation flow. It asks what type of doctor they are looking for rather than giving them the provider’s phone number as requested.

In this case it may be beneficial to setup two different goals to:

1. Help users setup an appointment. Goal ID set as SCHEDULE_APPOINTMENTS with example phrases like “Set up an appointment”, “See a doctor”, “Cardiologist appointment”.
2. Answer general provider questions. Goal ID set as PROVIDER_QUESTIONS with example phrases like “Doctor phone numbers”, “Do your providers accept cigna”, “What locations do you have?”.

Goal Completion

Within the log details for a user message, you will see the instructions that have been executed to get to the bot’s response. If you notice that within a conversation that a bot is unable to execute all of the instructions to complete a goal flow then you may need to update the setup of the instructions. This can happen in cases where:

• The user is unaware of how to respond in order to get to the next instruction. You can include more direction on how the user should proceed within the instruction details.
• The user is not interested in any of the options given (e.g., in a select step). Tell the bot to provide backup options or add a relevant backup instruction for the user to action on.

Search Configuration and Content

If a goal with a search instruction is triggered, but the bot is unable to respond then you can update either your linked Search configuration or update data in Yext Content.

Under the Yext Search section, you will want to check the following:

• The Results Count. If there are results for the predicted query but the bot doesn’t feel like the information is relevant to the users query (see the detectResultContent prompt in the reply instruction) then you may need to add or update entities in Content. The same may be the case if there are no results, but you might want to check the query ID or search query to further debug.

• The Search Query. Click the search query link to be directed to Test Search with the predicted query populated. When viewing this screen there are a few things to consider:

  • If you know the information exists in Content, but it isn’t appearing in Search you may need to update the Search configuration. Add or update searchable fields (i.e., adjust the algorithm used).
  • The Test Search screen defaults to the staging version of your experience. In most cases, your bot’s search instruction will be set to the production version. If you are seeing the proper results show up in the Test Search screen for staging but not in Chat, make sure to update the Search configuration label for the production version.

• Query ID. This link will direct you to the Search Log Details for the predicted query. Here you can further dive into the factors that influenced the search results. See the Debug the Backend unit for more details.

Using Intent Insights to Debug

In the Intent Insights unit, you learned that the Intent Insights screen displays data pertaining to clusters of similarly asked queries or common topics rather than at the individual query level. From this screen, you can see which topics are asked most frequently so you can tackle what’s most important to your customers.

Any topics that have the wand icon next to them, also have Suggested Actions associated with them. The Suggested Actions modal will guide you through actions you need to take if users queries aren’t being answered. They similarly follow the recommendations described below.

The key information on this screen to determine the performance of your bot is the goal match rate and the data match rate.

Improve Goal Matches

If clusters don’t have a 100% goal match rate, this indicates that some queries within that cluster don’t have a goal match and therefore the bot doesn’t know what actions to take to answer the user. To resolve this you may need to do one of two things:

1. Update the configuration of your goal. If you have an existing goal that should address a specific topic then you may need to add more example phrases. This helps the bot know what types of queries to look out for.
2. Add a new goal. If the existing goals in your bot configuration do not address the topic fully, then you will need to add a new goal to the configuration via the goals screen.

Improve Data Matches

If clusters are matched to a goal but don’t have a 100% data match rate, this means that the bot knows what actions to execute but doesn’t have the proper information to respond. Some things to address with a lower data match rate are:

1. Add or update the information being pulled (i.e., from Content or other data sources). If the information the bot needs to respond doesn’t exist in the data sources it’s pulling from then the fix may be as simple as adding it! For example, you can add new entities or update fields of existing entities in Content.
2. Modify your Search configuration. If you have the information living in Content but it is not being returned in Search then you will need to update the configuration. If this a new topic to Search then you can add a new vertical to include new entity types. If the proper entity types are included in Search already then you may need to include more searchable fields or leverage different Search algorithms.
3. Update the instruction settings within a goal. If you have relevant information in your data source and the data source is configured properly but the bot still can’t find a data match then you should check your goal’s instructions. The properties within the instructions might need to be modified. For example, a search instruction may need to include more verticals or a REST API instruction could have incomplete URLs or query parameters.