TLDR: Airbnb has developed and implemented two novel online evaluation methods, interleaving and counterfactual evaluation, to significantly speed up the testing of search ranking algorithm changes. Traditional A/B testing is slow for platforms with infrequent, high-stakes user actions like booking accommodations. Interleaving, using a ‘competitive pair’ design, allows for rapid comparison of two rankers by showing a user blended results, achieving up to 50X speedup. Counterfactual evaluation, a hybrid approach, generates results from both rankers but only shows one, using the ‘what-if’ scenario to derive highly sensitive metrics, demonstrating up to 100X speedup. Both methods are validated for consistency with A/B tests and enhance Airbnb’s ability to innovate faster.
Evaluating changes to search and recommendation systems is crucial for online platforms like Airbnb. These evaluations help create user-friendly features that drive business success. While randomized controlled experiments, commonly known as A/B tests, are a powerful tool, they present unique challenges for businesses, especially when dealing with significant purchases like booking accommodations.
A major hurdle with traditional A/B testing is the time it takes to achieve sufficient statistical power for conversion-based metrics. For Airbnb, where users might only travel a couple of times a year and spend days searching for the perfect stay, the traffic for experiments is lower, and the feedback (a booking) is delayed. As the system matures, the impact of individual innovations tends to be smaller, requiring even longer test durations to detect meaningful changes.
Offline evaluations, while quicker and more cost-effective, often lack accuracy because they can only assess what was shown by the existing system, not all potential options. This can lead to selection bias and a disconnect from actual online business metrics like conversion rates.
To bridge this gap, Airbnb developed and implemented two innovative online evaluation methods: interleaving and counterfactual evaluation. These approaches aim to provide rapid online assessments, helping identify the most promising candidates for A/B tests much faster.
Interleaving: A Faster Way to Compare
Interleaving experiments, first proposed years ago, involve presenting the same user with results from two different ranking algorithms (a control and a new treatment) simultaneously. By observing user actions, the system infers preferences between the two. Airbnb’s specific innovation in interleaving is a “competitive pair” based design. Instead of a complex merging process, it drafts items from both rankers, forming pairs. If the items are different, they are presented in an order determined by a coin flip, ensuring fairness. If they are the same, they are shown without a team assignment. User preferences are then determined by which ranker’s items lead to more “wins” within these competitive pairs.
This method has proven highly sensitive, offering up to a 50X speedup compared to traditional A/B testing in production. This means experiments can conclude much faster, using a smaller fraction of traffic. For instance, a test ranker that negatively impacted conversion could be detected in just one day’s data with interleaving, a process that would take weeks with A/B testing.
Counterfactual Evaluation: Addressing Interleaving’s Limitations
While powerful, interleaving has limitations. It can disrupt user experience when rankers use extensive set-level optimizations (like improving diversity) or when the search list influences other parts of the website (like a map view). It’s also less straightforward for continuous value outcomes like revenue.
Counterfactual evaluation addresses these issues by acting as a hybrid approach. Similar to interleaving, it generates results from both the control and treatment rankers for each search query. However, it only shows one of these results to the user, decided by a coin flip. The unshown result is the “counterfactual.” The evaluation then uses both the shown and counterfactual results to compute more sensitive metrics without altering the user’s experience.
This method leverages concepts like “direct decomposition” and “estimated reward based estimators” to analyze the impact. For example, it can assess how much better a listing would have ranked under the alternative algorithm. This approach has shown even greater sensitivity, with metrics demonstrating up to a 100X speedup compared to A/B testing.
Also Read:
- Unlocking AI’s Potential: A New Approach to Self-Evolving Agents
- RL-PLUS: A New Approach to Expand LLM Reasoning Capabilities Beyond Current Limits
Practical Insights and Future Directions
Both interleaving and counterfactual evaluation have been extensively used at Airbnb, conducting hundreds of experiments. They have significantly increased the capacity to test new ideas and improved the success rates of subsequent A/B tests. The choice between the two depends on the specific use case: interleaving offers high sensitivity with straightforward credit computation, while counterfactual evaluation is more robust for rankers with complex set-level optimizations.
These techniques are not limited to Airbnb and can be easily adopted by other online platforms, especially those where traffic or conversion events are limited, making traditional A/B testing time-consuming. The paper, titled Harnessing the Power of Interleaving and Counterfactual Evaluation for Airbnb Search Ranking, provides detailed insights into their implementation and validation.
Future work aims to further improve the accuracy of predicting outcomes from counterfactual results by incorporating real-time data collected during the experiment, refining the models for even greater sensitivity.
