TLDR: This research introduces a two-stage framework to enhance e-commerce search relevance. First, a powerful, domain-adapted Large Language Model (LLM) is built through continuous pre-training, supervised fine-tuning, and preference optimization to generate accurate relevance labels and detailed reasoning paths. Second, a novel Contrastive Reasoning Self-Distillation (CRSD) method transfers this reasoning capability to a lightweight BERT model, allowing it to implicitly apply complex logic without needing explicit reasoning paths during inference. This approach significantly improves search relevance accuracy and key business metrics in real-world e-commerce systems.
In the fast-paced world of e-commerce, connecting users with the right products or services quickly and accurately is paramount. This challenge is at the heart of search relevance prediction. While powerful Large Language Models (LLMs) have shown incredible capabilities in understanding and reasoning, their sheer size and computational demands make them too slow and costly for real-time e-commerce search systems. On the other hand, lightweight models like BERT are fast but often lack the deep reasoning abilities needed for complex semantic understanding.
A recent research paper, From Reasoning LLMs to BERT: A Two-Stage Distillation Framework for Search Relevance, introduces an innovative solution to bridge this gap. Authored by Runze Xia, Yupeng Ji, Yuxi Zhou, Haodong Liu, Teng Zhang, and Piji Li, this work proposes a two-stage framework that effectively transfers the advanced reasoning skills of a large LLM to a smaller, more efficient BERT model, making it suitable for industrial deployment.
Stage One: Building a Smart, Domain-Specific Teacher LLM
The first stage focuses on creating a highly capable ‘teacher’ LLM that is specifically adapted for e-commerce. General-purpose LLMs often lack the specific knowledge about products, services, and business rules unique to a particular platform. To overcome this, the researchers developed a three-step training pipeline:
- Domain-Adaptive Pre-training: An open-source LLM was continuously pre-trained on a massive dataset of 7 million domain-specific examples from search logs. This injected crucial platform knowledge into the model.
- Supervised Fine-tuning: The pre-trained LLM was then fine-tuned on a high-quality dataset of over 45,000 human-annotated examples. These examples included not just the correct relevance labels but also detailed ‘reasoning chains’ – the logical steps an expert would take to arrive at a decision. This taught the LLM to think like a human expert.
- Preference Optimization: To ensure the reasoning was not only correct but also reliable and aligned with platform preferences, a multi-dimensional reward model was introduced. This model evaluated the reasoning process itself, not just the final answer, providing richer feedback to further refine the LLM’s decision-making through reinforcement learning. This resulted in a powerful teacher LLM capable of generating accurate relevance labels and detailed, trustworthy reasoning paths for millions of query-service pairs.
Stage Two: Distilling Reasoning into a Lightweight Student Model
With the powerful teacher LLM in place, the second stage focuses on transferring its knowledge to a lightweight 6-layer BERT model, which is fast enough for real-time use. Traditional methods struggle with the architectural differences between LLMs and BERT. To address this, the researchers developed a novel method called Contrastive Reasoning Self-Distillation (CRSD).
CRSD works by having the same BERT model act as both a ‘teacher’ and a ‘student’ to itself, but under different input conditions:
- Teacher Configuration: The BERT model receives a ‘reasoning-augmented’ input, which includes the query, service, and the detailed reasoning path generated by the LLM.
- Student Configuration: The BERT model receives only the standard input – the query and service.
The core idea is to make the student configuration’s output similar to the teacher configuration’s output. This is achieved through a contrastive learning mechanism. By pulling the representations of the standard input closer to those of the reasoning-augmented input within the same batch, the BERT model is compelled to internalize the reasoning patterns. This means that even when it only sees the standard query and service during inference, it can implicitly apply the complex logic it learned from the LLM’s reasoning paths.
Also Read:
- Enhancing E-commerce Recommendations for Infrequent Shoppers with AI User Grouping
- Adapting Knowledge Distillation for Efficient Large Language Models
Real-World Impact and Validation
The effectiveness of this two-stage framework was rigorously tested. Offline evaluations on a large, real-world dataset showed significant improvements in relevance prediction accuracy. Crucially, these gains were validated through online A/B testing within Meituan’s search advertising system. The CRSD-enhanced model demonstrated substantial improvements across key business metrics, including a +0.64% increase in Ad Click-Through Rate (AdCTR) and a +1.73% increase in Ad Conversion Rate (AdCVR). A manual analysis also revealed a 0.8 percentage point reduction in relevance misjudgments.
This research offers a practical and effective way to integrate the advanced reasoning capabilities of LLMs into production-grade systems, overcoming the challenges of latency and domain specificity. It provides a clear path for e-commerce platforms to enhance search relevance, improve user experience, and drive business growth by making their search engines smarter and more efficient.
