TLDR: GReF is a new generative framework for reranking items in recommendation systems. It addresses challenges in existing methods by enabling end-to-end training and significantly improving inference efficiency. The framework uses a Gen-Reranker model, pre-trained on item exposure order and post-trained with user preferences via Rerank-DPO. Its key innovation, Ordered Multi-token Prediction (OMTP), allows it to generate multiple items simultaneously, making it fast enough for real-time applications. Experiments show GReF outperforms state-of-the-art methods in recommendation quality and has been successfully deployed in a real-world video app, Kuaishou, leading to substantial improvements in user engagement metrics.
In the complex world of multi-stage recommendation systems, where countless items vie for user attention, a critical step known as reranking plays a pivotal role. This process is all about refining an initial list of recommendations by considering how items relate to each other within that list, ultimately aiming to present the most appealing sequence to the user. However, current reranking methods often grapple with two significant hurdles: the difficulty of training the entire system from start to finish, and the slow speed of generating recommendations, especially with models that predict items one after another.
A new research paper, GReF: A Unified Generative Framework for Efficient Reranking via Ordered Multi-token Prediction, introduces an innovative solution to these challenges. Authored by Zhijie Lin, Zhuofeng Li, Chenglei Dai, Wentian Bao, Shuai Lin, Enyun Yu, Haoxiang Zhang, and Liang Zhao, this work proposes a unified framework designed to make reranking both more effective and much faster.
Understanding the Core Problem
Traditional reranking often involves a two-stage approach: a ‘generator’ proposes several possible sequences of items, and an ‘evaluator’ then picks the best one. While this seems logical, it creates a disconnect. The generator and evaluator are trained separately, making it hard to optimize the entire system cohesively. Furthermore, these generators often work like a conversation, predicting one item, then the next, and so on. This ‘autoregressive’ nature means that as the list of recommendations gets longer, the time it takes to generate it increases, making it impractical for real-time applications like short-video apps.
Introducing GReF: A Unified Approach
GReF, or the Unified Generative Efficient Reranking Framework, tackles these issues head-on. At its heart is the ‘Gen-Reranker’, an autoregressive model that combines a bidirectional encoder and a dynamic autoregressive decoder. This allows it to generate recommendation sequences that capture the natural flow of user preferences. Instead of considering billions of items, it dynamically matches generated item representations with a smaller set of candidate items, making the process much more efficient.
A Smart Training Strategy
GReF employs a sophisticated two-stage training process:
-
Pre-training on World Knowledge: Unlike models that only learn from explicit user feedback (which can be sparse), GReF first undergoes pre-training on the ‘item exposure order’ from existing recommendation systems. This means it learns from the actual sequences of items users were shown, even if they didn’t interact with every single one. This broad exposure provides the model with a high-quality initial understanding of user interests and general recommendation patterns, similar to how large language models learn from vast amounts of text.
-
Post-training with User Preferences (Rerank-DPO): After pre-training, GReF is fine-tuned using a method called Rerank-DPO (Direct Preference Optimization). This stage directly integrates user preferences. It constructs pairs of ‘winning’ and ‘losing’ sequences based on user feedback (like clicks) and the original item positions. By comparing these pairs, the model learns to prioritize sequences that users are more likely to prefer, without needing a separate evaluator, thus enabling true end-to-end optimization.
Boosting Efficiency with Ordered Multi-token Prediction (OMTP)
To overcome the speed limitations of traditional autoregressive models, GReF introduces Ordered Multi-token Prediction (OMTP). Instead of predicting one item at a time, OMTP trains the Gen-Reranker to simultaneously generate multiple future items in a single step, while still ensuring their correct order. This significantly reduces the number of steps required to generate a full recommendation list, making GReF fast enough for real-time deployment in high-traffic applications.
Real-World Impact and Performance
The effectiveness of GReF was rigorously tested through extensive offline experiments on both a public dataset (Avito) and a large industrial dataset from Kuaishou, a popular short-video app with over 300 million daily active users. GReF consistently outperformed state-of-the-art reranking methods in terms of recommendation quality metrics like AUC and NDCG.
Crucially, GReF achieved this superior performance with an inference time (the time it takes to generate recommendations) that was nearly comparable to much simpler, non-autoregressive models. For instance, on the Kuaishou dataset, GReF’s inference time was 12.97 ms, significantly faster than Seq2Slate (67.34 ms) and very close to NAR4Rec (12.67 ms), a non-autoregressive model. This efficiency is a direct result of the OMTP technique.
Furthermore, GReF was deployed in a real-world A/B test on the Kuaishou app, involving 8% of its massive user base over one week. The results were remarkable, showing significant improvements across key online metrics:
-
Views: +0.33%
-
Long Views: +0.42%
-
Likes: +1.19%
-
Forwards (Shares): +2.98%
-
Comments: +1.78%
These improvements highlight GReF’s ability to not only enhance content visibility and engagement but also foster a more interactive and participatory user experience.
Also Read:
- Dynamic Filters for Smarter Recommendations: Introducing TV-Rec
- TOOLRM: Enhancing Language Models’ Tool-Use with Efficient Reasoning and Generative Rewards
Conclusion
GReF represents a significant advancement in reranking for recommendation systems. By unifying the generative process with end-to-end training and introducing efficient multi-token prediction, it addresses long-standing challenges of training complexity and inference speed. Its successful deployment in a major real-world application like Kuaishou demonstrates its practical value and potential to deliver more personalized and engaging user experiences.
