Version Comparison| Hitchhikers Platform

What You’ll Learn

In this section, you will learn:

  • What Version Comparison is & why it’s important
  • How to use the Version Comparison feature
  • When to use Version Comparison

What is “Version Comparison”

Versioning our search configuration in Answers helps us manage changes over time. Every time we save an update to the configuration, a new version is created. The first time you save creates version 1, and each subsequent change you save increments the version number. We learned about this system of version labels in the “Configuration Labels” unit.

And the versions add up! It’s not long before you find yourself on version 100+!

So what happens if you make an adjustment (for example, you change an NLP filter to a textSearch) - but you want to see what effects this has on Search Quality. If only there was some way we could compare results between 2 different configuration versions….

Enter, Version Comparison!! Version Comparison allows us to see how two result sets for a given group of Search Terms compare to each other based on the configuration change (or changes).

Why is it important to use Version Comparison?

Configuration Changes are the primary way a Hitchhiker can impact the Search Quality of an Answers Experience. It’s important to understand and have visibility into how these changes influence results. Did the change improve the results, or did it make the results worse (often referred to as a regression).

This becomes particularly important if your Answers Experience is already live, and you want to vet updates before pushing them publicly to your website. You wouldn’t want to publish a regression and expose users to a poor search experience.

Versioning ensures we can safely play around with configuration updates, and Version Comparison helps us confidently assess the effectiveness of those changes.

Getting Started

There is a tab in the Answers > Experiences screen on the left hand sidebar called Version Comparison. This is where you’ll get started.

Version Comparison provides you with three different ways to assess the changes you’ve made on your experience. We call these:

  1. Version Comparison Summary
  2. Version Comparison Detail
  3. Search Comparison

(listed in order of descending granularity)

Version Comparison Summary

With Version Comparison Summary, you can see things like:

  • Configuration Versions Compared
  • # of Search Terms Changed
  • Quality Metrics

This is also the screen where you can initiate a new Version Comparison run!

Version comparison

Version Comparison Detail

The next level down from Version Comparison Summary is the Version Comparison Detail screen. You can navigate to this screen by selecting View Details on the summary screen. Here you can view the impact of your changes on a Search Term level with additional metrics like:

  • Adds
  • Removes
  • Total Differences
  • Quality Metrics (these are only available when using Benchmarked Search Terms)
    • Answer@1
    • Precision@1
    • Precision@3

This helps you understand which Search Terms were impacted and whether those changes resulted in an improvement or a regression.

Version Comaprison Detail

Search Comparison

If you want to investigate the impacts of your Search Configuration change at the lowest and most granular level, then Search Comparison is your answer! You can navigate to this screen by selecting a specific search term from the Version Comparison Details screen.

With Search Comparison, you can see a side by side comparison of the results returned by Search Term. This allows you to see exactly what changed between the two Configurations. Search Comparison will return a recreation of the results for the Search Term, along with helpful metadata about the Search and Search Factors.

Search Comparison

Running a Version Comparison

To run a new version comparison, select Run New Comparison. From there you’ll be asked to indicate the two versions you’d like to compare and the set of Search Terms you’d like to run the comparison on.

Run new Version Comparison

With Version Comparison, you can select from any one of the following groups of Search Terms to run your comparison:

  1. Saved Search Terms
  2. Popular Search Terms
  3. Benchmarked Search Terms
  4. Labeled Search Terms

Once you click Run Comparison , a Version Comparison will be triggered and your Version Comparison will be marked In Progress on the Summary screen and will update to Completed when it has finished running.

When should I be comparing versions?

It’s NOT necessary to run a comparison for every update that is made to your config. Generally, Version comparisons should be run for major updates that might change multiple results. Minor changes don’t require a full Version Comparison.

Examples of major changes might include:

  • Addition of a new vertical
  • Changing an nlpFilter -> TextSearch
  • Removing highly leveraged filter criteria (ex: removing “builtin.location” search)
  • Adding highly leveraged filter criteria (ex: adding “builtin.location” search)

Examples of minor changes might include:

  • Adding a synonym
  • Adding a saved search filter
  • Adding facets
unit Quiz
+20 points
Daily Quiz Streak Daily Quiz Streak: 0
Quiz Accuracy Streak Quiz Accuracy Streak: 0
    Error Success Question 1 of 3

    Why is Version Comparison important?

    Error Success Question 2 of 3

    Which of the following are Search Configuration changes that should prompt you to run a version comparison?

    Error Success Question 3 of 3

    How granular can I get when comparing Search Configuration versions?

    Wahoo - you did it! 🙌

    You've already completed this quiz, so you can't earn more points.You completed this quiz in 1 attempt and earned 0 points! Feel free to review your answers and move on when you're ready.
1st attempt
0 incorrect
Splash Loading