TLDR: A new research paper introduces Rejuvenated Cross-Entropy for Knowledge Distillation (RCE-KD), a novel method that enhances the effectiveness of Cross-Entropy (CE) loss in training smaller, efficient recommender systems. The paper analyzes why traditional CE loss struggles in this domain due to its reliance on an ‘assumption of closure’ that is often unmet when distilling rankings from a teacher model to a student model on a subset of items. RCE-KD addresses this by splitting the teacher’s top items into two subsets, applying tailored loss functions with an adaptive sampling strategy for each, and fusing them with a dynamic weighting scheme. This approach significantly improves recommendation accuracy while maintaining training efficiency, as demonstrated through extensive experiments.
Recommender systems are at the heart of many online experiences, suggesting everything from movies to products. As these systems grow more sophisticated, they often rely on increasingly large and complex models to achieve better accuracy. However, these massive models come with significant drawbacks: high storage costs, slow response times, and increased maintenance expenses, which can ultimately lead to lower user satisfaction.
To tackle this challenge, a technique called Knowledge Distillation (KD) has emerged as a promising solution. KD aims to transfer the ‘knowledge’ from a large, powerful model (the ‘teacher’) to a smaller, more efficient one (the ‘student’). Once the student model is trained, it can be used for recommendations, offering similar accuracy with much lower operational costs. A popular method within KD is response-based distillation, where the student tries to mimic the teacher’s predictions. Cross-Entropy (CE) loss is a widely used function for this purpose in fields like computer vision and natural language processing.
However, applying CE loss directly to recommender systems for KD presents unique challenges. Recommender systems primarily focus on ranking items, especially those a user is most likely to prefer, rather than just predicting specific scores. Additionally, KD in this context can only be performed on a small subset of items due to the enormous number of items available. This paper, titled “REJUVENATING CROSS-ENTROPY LOSS IN KNOWLEDGE DISTILLATION FOR RECOMMENDER SYSTEMS,” delves into these issues.
The researchers, Zhangchi Zhu and Wei Zhang from East China Normal University, analyze the connection between CE loss and a key ranking metric called Normalized Discounted Cumulative Gain (NDCG). They theoretically prove that minimizing CE loss can maximize a lower bound of NDCG, but this holds true only if an important condition, termed the ‘assumption of closure,’ is met. This assumption requires that the subset of items used for distillation must include all items that the student model ranks higher than any item in that subset. The problem is that the goal of KD is to distill the rankings of the teacher’s top items, but the student’s top items often differ significantly from the teacher’s, especially early in training. This ‘vast gap’ makes it difficult for the teacher’s top items alone to satisfy the closure assumption, leading to poor performance for vanilla CE loss.
To bridge this gap and unlock the full potential of CE loss in recommender system KD, the authors propose a novel approach called Rejuvenated Cross-Entropy for Knowledge Distillation (RCE-KD). This method introduces several key innovations:
Splitting Teacher’s Top Items
RCE-KD begins by dividing the teacher’s top-ranked items into two distinct subsets. The first subset consists of items that are highly ranked by both the teacher and the student. The second subset includes the remaining teacher’s top items that the student does not rank highly.
Tailored Loss Functions for Each Subset
For the first subset (items highly ranked by both), RCE-KD directly applies CE loss on the student’s top items. This is effective because the student’s top items naturally satisfy the assumption of closure, allowing for precise distillation of fine-grained rankings.
For the second subset (teacher’s top items not well-ranked by the student), a clever sampling strategy is introduced. This strategy adaptively samples additional items, primarily from the student’s highly-ranked items, to create a new item set. CE loss is then computed on this expanded set, which is designed to approximately satisfy the assumption of closure. This helps the student improve its rankings for those items it initially struggled with.
Also Read:
- Disentangling Information for Smarter Recommendations
- Finding the Right Dose: A Scaling Law for Knowledge Infusion in LLMs
Adaptive Loss Fusion
Finally, RCE-KD combines the losses from these two subsets using an adaptive weighting scheme. This dynamic weighting adjusts based on the overlap between the teacher’s and student’s top items. If there’s little overlap, more weight is given to the loss that encourages the student to align with the teacher’s top items. If there’s significant overlap, more weight is given to the loss that refines the student’s rankings.
Extensive experiments were conducted on three public datasets (CiteULike, Gowalla, and Yelp) across various teacher-student model configurations, including both homogeneous (same model type) and heterogeneous (different model types) settings. The results consistently demonstrated that RCE-KD significantly outperforms existing knowledge distillation methods. Furthermore, the method maintains high training efficiency, comparable to simpler baselines, by optimizing the sampling process.
This research provides a crucial theoretical understanding of CE loss in the practical context of knowledge distillation for recommender systems. By addressing the limitations of traditional CE loss through its innovative split-and-fusion paradigm, RCE-KD offers a powerful and efficient way to train smaller, high-performing recommendation models. You can find the full research paper here.
