TLDR: HiLoMoE is a new framework for Click-Through Rate (CTR) prediction models that combines hierarchical routing with lightweight LoRA experts. It enables efficient scaling both horizontally (more experts) and vertically (more layers) while maintaining parameter efficiency and parallel inference. A three-stage training process ensures stability and expert diversity, leading to improved prediction accuracy and reduced computational cost compared to existing methods.
Deep learning models have become indispensable in predicting click-through rates (CTR), a fundamental task in modern recommender systems. These systems are crucial for ranking items, allocating ad impressions, and personalizing content delivery. However, as user bases and system scales continue to expand, there’s a growing need for more expressive and efficient CTR models that can deliver accurate predictions without excessive latency or resource consumption.
Traditionally, model scaling has taken two main paths: “vertical scaling” by stacking more layers, and “horizontal scaling” through Mixture of Experts (MoE). While deeper networks (vertical scaling) can capture complex representations, they often lead to slow, sequential computations and a large number of parameters. On the other hand, MoE (horizontal scaling) improves efficiency by activating only a small subset of specialized “experts” for each input, allowing for personalized computation. However, conventional flat MoE layers often struggle to capture the inherent hierarchical structure found in recommendation tasks, such as categories, products, and brands.
Introducing HiLoMoE: A Holistic Scaling Framework
To address these challenges, researchers from the University of Illinois Urbana-Champaign and Meta AI have proposed HiLoMoE, a hierarchical LoRA Mixture of Experts framework. This innovative approach combines the strengths of both vertical and horizontal scaling in a parameter-efficient manner, aiming to push the boundaries of performance and efficiency in CTR prediction.
HiLoMoE is built upon three key innovations:
- LoRA Experts: For efficient horizontal scaling, each expert is designed as a lightweight, rank-1 perturbation of a shared base weight matrix. This significantly reduces parameter and memory overhead, allowing for a large number of experts without inflating the overall model size.
- Hierarchical Routing: To enable efficient vertical scaling, HiLoMoE introduces a routing mechanism where expert selection at each layer is conditioned on the routing scores from previous layers. This creates combinatorially diverse expert compositions. Crucially, this design allows all MoE layers to execute in parallel during inference, as routing decisions depend only on prior layer scores, not intermediate expert outputs. This means vertical scaling adds no additional inference cost.
- Principled Three-Stage Training Framework: To ensure stable optimization and promote expert diversity, HiLoMoE employs a progressive training pipeline. This framework includes a backbone warmup, sequential expert warmup, and a final full-model fine-tuning stage, augmented with auxiliary losses to prevent expert collapse and ensure balanced utilization.
Why Hierarchical Structure Matters
Recommendation tasks often involve information with a natural hierarchy. For instance, an item might belong to a broad “electronics” category, then a “laptop” product type, and finally a specific “Apple” brand. Traditional single-layer MoE models find it difficult to disentangle these multi-level semantics, leading to less specialized experts. HiLoMoE’s hierarchical routing directly addresses this by allowing earlier layers to focus on high-level attributes and later layers to refine decisions based on finer-grained features, leading to more accurate and personalized recommendations.
Performance and Efficiency Gains
Extensive experiments on four public datasets (AmazonElectronics, TaobaoAd, KuaiVideo, and MicroVideo) demonstrate HiLoMoE’s effectiveness. Compared to non-MoE baselines, HiLoMoE consistently achieved an average AUC improvement of 0.20% and an 18.5% reduction in FLOPs (floating-point operations per second), indicating a superior performance-efficiency trade-off. It also outperformed existing state-of-the-art MoE baselines like Switch Transformer, MoLE, and HydraLoRA, showing better accuracy with fewer parameters.
The framework also exhibits promising scalability. Increasing the number of experts generally improves performance, confirming the benefits of horizontal scaling. Modest vertical scaling (adding more MoE layers) also leads to incremental improvements, especially in datasets with longer behavioral sequences.
Also Read:
- HoMer: A Unified Approach to Enhance Click-Through Rate Prediction in Recommender Systems
- MC#: A Dual Approach to Compress Mixture-of-Experts AI Models
Robust Training for Complex Models
The three-stage training framework plays a vital role in HiLoMoE’s stability and performance. Direct training of complex MoE models can be unstable. The progressive warmup strategy, starting with a backbone model and then gradually introducing and fine-tuning experts, ensures a stable optimization process. Additionally, auxiliary losses are used to encourage balanced expert utilization, preventing a scenario where only a few experts are consistently activated, which would limit the model’s capacity. These losses are carefully applied to only update router parameters, avoiding interference with the main prediction task.
In conclusion, HiLoMoE represents a significant step forward in designing scalable and efficient CTR prediction models. By intelligently combining low-rank expert parameterization with hierarchical routing and a robust training framework, it offers a powerful solution for enhancing recommendation systems under real-world efficiency constraints.
