TLDR: This research paper presents a comprehensive, data-centric analysis of large-scale Mixture-of-Experts (MoE) LLMs, revealing predictable patterns in expert selection despite their apparent randomness. By profiling models up to 671 billion parameters across diverse workloads, the authors identified six key temporal and spatial insights. These insights enable the design of more efficient serving systems, demonstrated through a case study on wafer-scale GPUs where architectural modifications leveraging these predictions achieved significant speedups (4.0x-6.3x) by optimizing task allocation and memory management to reduce data movement overhead.
Large Language Models (LLMs) with a Mixture of Experts (MoE) architecture are becoming increasingly popular for their ability to achieve impressive performance while managing computational costs. However, a significant challenge in serving these models, especially in large-scale systems, is the substantial overhead caused by data movement. This overhead arises because MoE models dynamically route each request to a subset of experts, and this selection process often appears random, leading to frequent and costly data transfers across different parts of a system.
A new research paper titled “Orders in Chaos: Enhancing Large-Scale MoE LLM Serving with Data Movement Forecasting” delves into this critical issue. The authors, Zhongkai Yu, Yue Guan, Zihao Yu, Chenyang Zhou, Shuyi Pei, Yangwook Kang, Yufei Ding, and Po-An Tsai, conducted an extensive study to understand and predict these data movement patterns. Their work provides the first comprehensive data-centric analysis of MoE models at scale, offering crucial insights to guide the design of future serving systems.
Understanding the MoE Challenge
Traditional LLMs use all their parameters for every request. MoE models, on the other hand, employ multiple “experts” and a “gating mechanism” to select only a few experts for each request. This allows models to scale to massive sizes without a proportional increase in computational cost per request. However, as MoE models grow larger and sparser, the dynamic and seemingly random nature of expert selection leads to a bottleneck: data movement. When an expert is needed but not available on the local processing unit, it must be fetched from a remote location, incurring significant latency and consuming valuable bandwidth.
The paper highlights two main aspects of this data movement challenge. From a temporal perspective, the sheer number of possible expert combinations makes it seem impossible to predict which experts will be needed next, hindering prefetching or caching strategies. Spatially, if expert selection were truly random, it would cause severe workload imbalances, leaving many processing units idle while others are overloaded.
Uncovering Predictable Patterns
Fortunately, the researchers discovered that MoE expert selections are not entirely unpredictable. They performed a comprehensive, data-movement-centric profiling across three state-of-the-art large-scale MoE models (DeepSeek V3, Llama4, and Qwen3), ranging from 235 billion to 671 billion parameters. This involved analyzing over 24,000 requests across diverse workloads, generating more than 150GB of trace files. Through systematic analysis from both temporal (time-dependent) and spatial (distribution) perspectives, they distilled six key insights.
Key Insights for MoE Serving Systems
The insights are categorized into temporal and spatial relations.
Temporal Relations: These capture time-dependent patterns, useful for optimizing data movement within individual processing units. One key insight is prefill-data-driven prediction, where expert selections during the initial “prefill” stage strongly predict selections during the subsequent “decode” stage. This allows systems to anticipate future expert needs. Another insight is cross-hierarchy memory management, where expert selections show correlations at both layer-level (between adjacent model layers) and token-level (between consecutive tokens in the same layer). These patterns can be leveraged for dynamic prefetching and caching across different memory tiers, like faster caches for short-term needs and larger DRAM for longer-term needs.
Spatial Relations: These reflect expert distribution and imbalance across the system, enabling strategies for system-wide expert placement and workload distribution. An important insight is expert-placement-aware workload distribution, meaning workload allocation should consider where experts are physically located to minimize data movement and balance load. Another is popular expert decentralization, recognizing that some experts are activated far more frequently than others. Duplicating or distributing these “hot” experts across multiple compute units can balance the workload. Expert-pair separation is also crucial, as certain pairs of experts are frequently activated together. Assigning these co-activated pairs to different compute units can maximize parallelism, though this needs to be balanced against communication overhead. Finally, workload-aware serving systems can leverage the fact that expert popularity varies significantly based on task type, such as biology vs. math, and even language, like English vs. Chinese. Systems can use this information to proactively migrate or redistribute experts before serving specific types of queries.
Also Read:
- Unlocking Multilingual Potential: Steering AI’s Language Experts
- SliceMoE: Enhancing AI Models Through Sub-Token Routing
Case Study: Enhancing Wafer-Scale GPUs
To demonstrate the practical applicability of their insights, the researchers conducted a case study on future wafer-scale GPU architectures. These advanced GPUs, moving towards multi-chiplet designs, offer immense computing power but face challenges with simplistic task allocation and inefficient memory management for MoE models.
Leveraging their insights, the team proposed two key innovations. First, an intelligent task allocation algorithm with a two-level command processor architecture. This system considers expert placement and selection patterns to dynamically assign tasks, minimizing inter-die communication and balancing workloads. Second, a data-driven predictor with hardware-managed High Bandwidth Memory (HBM). This system uses temporal insights to predict future expert needs, caching critical data locally to reduce communication overhead between different dies on the wafer.
These architectural modifications, even minor ones, achieved substantial performance gains, delivering average speedups of 6.3x on DeepSeek V3 and 4.0x on Qwen3 for MoE LLM serving throughput on wafer-scale GPUs. The paper’s findings and simulation framework are publicly available, encouraging further research in this area. You can find more details in the full research paper: Orders in Chaos: Enhancing Large-Scale MoE LLM Serving with Data Movement Forecasting.
This work represents a significant step towards more efficient and scalable serving of large-scale MoE LLMs, providing a model-centric foundation for optimizing systems across various platforms, from multi-GPU clusters to CXL-based memory disaggregation and future wafer-scale architectures.
