TLDR: A study by OTTO compared deep neural networks (DNNs) with traditional tree-based models (like LambdaMART) for e-commerce learning-to-rank tasks. Using a large proprietary dataset and an 8-week online A/B test, they found that a simple Two-Tower DNN architecture significantly improved total clicks and revenue, while maintaining units sold, outperforming their production-grade LambdaMART model. This suggests DNNs are a viable and effective alternative for industrial ranking systems.
In the fast-paced world of e-commerce, how products are ranked in search results and recommendations directly impacts user engagement and, ultimately, sales. For a long time, tree-based models, particularly Gradient-Boosted Decision Trees (GBDTs) like LambdaMART, have been the go-to solution for Learning-to-Rank (LTR) tasks, setting a high standard for effectiveness.
However, a persistent question in the industry has been whether more advanced deep neural networks (DNNs) can truly surpass these established tree-based powerhouses. While academic studies have sometimes pointed to the superiority of DNNs, particularly transformer-based architectures, industry practitioners have faced challenges. Issues like scalability and the lack of comprehensive online A/B test results on real-world e-commerce data have left many uncertain about adopting DNNs for their ranking systems.
Addressing this crucial uncertainty, researchers from OTTO, a major e-commerce company, conducted a thorough comparison. They systematically benchmarked various DNN architectures against their own production-grade LambdaMART model. This study is particularly significant because it used a large, proprietary dataset derived from OTTO’s actual user interaction logs and, critically, validated its findings through an extensive 8-week online A/B test.
The study explored three distinct DNN architectures: the Two-Tower model, the Cross-Encoder model, and the Transformer model. Each architecture offers different ways of processing product and context features, with the Two-Tower model being notable for its efficiency during inference (when the model is used to make predictions). They also evaluated two common loss functions, RankNet and Softmax Cross-Entropy, adapting them to handle scenarios where users interact with multiple products.
The offline experiments, conducted on a massive dataset of 43 million training samples and 700,000 test samples, showed promising results. All tested DNNs generally outperformed the traditional LambdaMART baseline in terms of NDCG for clicks (a metric reflecting ranking quality for user clicks) and Average Item Value (AIV), which serves as a proxy for revenue. While DNNs sometimes showed slight underperformance in NDCG for orders, some models achieved comparable results.
Notably, the simpler Two-Tower model, when trained with the Softmax Cross-Entropy loss, emerged as the strongest candidate. It improved click-based ranking quality and revenue potential without negatively impacting order-based ranking quality. Interestingly, the more complex Transformer model, which has shown strong results in other academic contexts, performed less effectively for order-based metrics on OTTO’s dataset.
The real validation came from the 8-week online A/B test conducted on OTTO’s live e-commerce platform. This real-world test directly compared the top-performing Two-Tower DNN model against the production LambdaMART system. The results were compelling: the DNN achieved a statistically significant increase of 1.86% in the total number of clicks and a 0.56% uplift in generated revenue. Crucially, the total units sold remained stable, indicating that the improvements in clicks and revenue were genuine gains rather than shifts in purchasing behavior.
Also Read:
- Enhancing Recommender Systems for New Items with Adaptive Content Embeddings
- Unlocking Deeper User Understanding: The LANTERN Model
While DNNs do incur higher training and serving costs, the study concluded that these costs were negligible when weighed against the significant performance gains observed. This research provides valuable, actionable insights for e-commerce practitioners. It demonstrates that deep learning approaches, especially a well-tuned, simpler architecture like the Two-Tower model, can be a highly effective alternative to traditional GBDT-based models in large-scale industrial ranking systems. For more detailed information, you can refer to the full research paper here.
