TLDR: MTmixAtt is a novel Mixture-of-Experts (MoE) architecture with Multi-Mix Attention designed for large-scale recommendation systems. It tackles challenges like manual feature engineering and limited cross-scenario transfer by automatically grouping heterogeneous features (AutoToken) and enabling efficient, scenario-aware feature interactions (MTmixAttBlock). Extensive experiments on Meituan’s industrial TRec dataset and online A/B tests demonstrate that MTmixAtt consistently outperforms state-of-the-art baselines, achieving significant improvements in CTR, CTCVR, and key business metrics like Payment PV and GTV across various recommendation scenarios.
In the fast-paced world of online platforms, recommender systems are the unsung heroes, guiding users to products, videos, and services they’ll love. However, building these systems for massive, diverse platforms like Meituan, a leading e-commerce and local-life service provider, comes with significant challenges. Traditional methods often rely on painstaking manual feature engineering and rigid, scenario-specific architectures, which makes it hard to adapt models across different parts of the platform or scale them effectively.
Addressing these hurdles, a team of researchers from Meituan and Beijing University of Posts and Telecommunications has introduced a groundbreaking solution called MTmixAtt. This innovative model, detailed in their paper “MTmixAtt: Integrating Mixture-of-Experts with Multi-Mix Attention for Large-Scale Recommendation”, offers a unified Mixture-of-Experts (MoE) architecture combined with Multi-Mix Attention, specifically designed for the complexities of large-scale recommendation tasks.
What Makes MTmixAtt Stand Out?
MTmixAtt is built on two core components that work in harmony to overcome the limitations of previous systems:
-
AutoToken Module: Imagine a system that can automatically understand and group different types of information – like user profiles, item details, and past interactions – into meaningful categories, without any human intervention. That’s what AutoToken does. It intelligently clusters heterogeneous features into semantically coherent “tokens,” eliminating the need for manual, often subjective, feature grouping. This data-driven approach ensures consistency and adaptability across various data distributions and scenario requirements.
-
MTmixAttBlock Module: This is where the magic of interaction happens. The MTmixAttBlock facilitates efficient communication between these automatically generated tokens. It uses a learnable mixing matrix, which dynamically captures complex relationships among feature groups. Furthermore, it integrates both shared dense experts and scenario-aware sparse experts. This clever combination allows the model to identify universal patterns that apply across all scenarios, while also capturing unique behaviors specific to individual scenarios, all within a single, cohesive framework.
A Unified and Scalable Solution
The beauty of MTmixAtt lies in its unified approach. It brings together feature grouping, the modeling of diverse feature types, and adaptation across multiple scenarios into one scalable framework. This means that a model optimized for one part of a platform, like the homepage, can more easily adapt to other areas, such as promotional campaigns, without requiring costly re-engineering.
Real-World Impact and Impressive Results
The researchers put MTmixAtt to the test using Meituan’s industrial TRec dataset, a massive collection of purchase logs from hundreds of millions of users. The results were compelling: MTmixAtt consistently outperformed state-of-the-art baselines, including advanced Transformer-based models, WuKong, HiFormer, MLP-Mixer, and RankMixer. It showed superior performance in key metrics like Click-Through Rate (CTR) and Click-to-Conversion Rate (CTCVR).
Even more impressively, MTmixAtt demonstrated excellent scalability. As the model size increased to MTmixAtt-1B (one billion parameters), it showed monotonic gains across all evaluation metrics, confirming its power-law scaling behavior, similar to what’s observed in large language models.
The true validation came from large-scale online A/B tests conducted in Meituan’s live environment. In the crucial Homepage scenario, MTmixAtt led to a significant increase in Payment Page Views (PV) by +3.62% and Actual Payment Gross Transaction Volume (GTV) by +2.54%. It also delivered comprehensive improvements across various other scenarios, such as Special Offer Groupon Feeds and Short Video recommendations. These real-world results underscore MTmixAtt’s ability to not only boost commercial outcomes but also enhance user experience by providing more relevant and diverse recommendations.
Also Read:
- Enhancing Dense Retrieval Models with a Single Mixture-of-Experts Block for Improved Efficiency and Generalization
- Boosting Recommendation Accuracy: A New Approach with Language Models and Collaborative Feedback
Looking Ahead
MTmixAtt represents a significant leap forward in the design of industrial recommender systems. By offering a unified, scalable, and adaptable solution for modeling heterogeneous features across diverse scenarios, it promises to improve both user engagement and business performance. The researchers plan to further explore more efficient scaling strategies, extend MTmixAtt to multimodal recommendation settings, and investigate its applicability to other industrial domains.
