Impact of Machine Learning Enhanced Reranking

A good search engine provides the most relevant results for a given query. Here at Yext we think semantic understanding is one of the most foundational components of relevance. If someone searches "how do I apply for a job" you have tons of information from the user to provide super relevant search results.

However, if someone searches for a short keyword or phrase, the semantic signal isn't as helpful. Let's take a specific example. Yext works with Chick-fil-A and they sell some amazing chicken products. If a user is searching their menu and searches for "chicken sandwich" what is the ideal result set?

Well, Chick-Fil-A has several types of chicken sandwiches. Semantically out of the box our search engine finds 8 potential menu items that fit the user's query. While it's pretty easy to find all the results that should be displayed, it's much harder to rank those results. The first 6 all seem like really good matches. The Chicken Biscuit is kind of a sandwich, but probably not as helpful, and the chicken noodle soup does contain chicken, but isn't a sandwich, so probably isn't relevant. However, between the first 6 items it's very difficult to know what the user is looking for. If the user searched for something more specific, like "grilled chicken sandwich", then semantics would give us all we need to know. But without that information, the semantic information is lacking.

This is where Dynamic Reranking comes in. Dynamic Reranking is a part of Yext Answers that uses machine-learning to rerank search results. Often this is referred to as a Learning to Rank algorithm where the goal is to rank results in an ideal order. With Dynamic Reranking, the system's goal is to minimize the average click position of a given search result.

Average click position is the average click position within a vertical for a given ranking. In the example, if the user clicked on "Spicy Chicken Sandwich", that would be the 3rd position. If they clicked on Chicken Noodle Soup, that would be the 8th position. With a grid layout, we rank positions left to right, top to bottom. With a more vertical layout, you would rank top to bottom.

In general, the lower the average click position (closer to 1), the better the search results. If a lot of people are clicking on an item that is pretty far down the page, that probably indicates that the search results ordering should be tweaked.

A naive way to do this would be to just rank the search results based on clicks. This approach is a mistake however, as users are more likely to just click on results at the top just because they are at top. Imagine the use case of a doctor finder. Users are most likely to click on the top doctor that shows up just because they are first.

In order to control for this bias, Yext uses an algorithm based on BubbleRank. BubbleRank (not to be confused with Bubble Sort, although conceptually similar) is an algorithm that re-ranks search results to optimize for click position. The goal of this algorithm is to find the ideal order of results. Under the hood, it uses a multi-armed bandit to run thousands of micro A/B tests to continually tweak results and improve ordering.

Chick-Fil-A uses Dynamic Reranking on their menu vertical, and it's been extremely successful. Since turning it on, the average click position has gone from 2.4 to 1.7. Given the minimum click position is 1, this reduction is a 50% increase! A huge improvement by just checking an option inside their answers config.

If you dive into the details, it gets even more interesting. Here are the top 20 search terms with average click position by week. You can see every term has an upward trend, and one term "Chicken Nuggets" went from an average click position of 6.2 to 1.5! It turns out the "grilled nuggets" option isn't as popular as the fried option!

Inside of the Yext Platform, you can also dive into an individual search and see exactly what happened. I searched for Chicken Nuggets, and in the query debugger, you can see the re-ranking in action:

You'll see the Chick-Fil-A Nuggets have risen to the top of the search results, even though the menu item doesn't include the term "chicken." This is the power of dynamic reranking!

Yext is constantly improving its Answers algorithms to help our clients increase conversions, answer important questions, reduce support costs, and gain granular insights into how their customers behave online. Interested in how Yext Answers can help your business? Please feel free to contact us here and a member of our team will help you get up and running in no time!