TLDR: A new method called Importance-Driven Expert Scheduling enables efficient deployment of Mixture-of-Experts (MoE) Large Language Models on edge devices with limited GPU memory. It achieves this by prioritizing high-importance experts for preloading and substituting low-importance active experts with similar ones already in GPU memory, significantly reducing latency and improving cache hit rates while maintaining accuracy.
Large Language Models (LLMs) are becoming increasingly vital for various applications, from smart homes to autonomous vehicles. Deploying these powerful models directly on edge devices – like your smartphone or a car’s onboard computer – offers significant advantages such as instant interaction, enhanced data privacy, and improved reliability without constant network reliance. However, a major hurdle is the limited memory and processing power of these consumer-grade edge devices, especially for complex LLM architectures like Mixture of Experts (MoE).
The Mixture of Experts (MoE) architecture is a clever way to scale LLMs. Instead of activating all parts of the model for every query, MoE models only activate a small subset of specialized ‘experts’ per input. This reduces the computational load, making them more suitable for resource-constrained environments. Yet, even with MoE, edge devices often can’t fit the entire model into their integrated GPU memory. This necessitates ‘expert offloading,’ where some experts are stored in the CPU memory and transferred to the GPU or processed directly on the CPU when needed.
The challenge arises because transferring data via PCIe (the connection between CPU and GPU) or computing on the CPU is significantly slower than processing on the GPU. This can lead to substantial delays, hindering the real-time performance crucial for edge applications. Existing offloading methods often treat all active experts uniformly, failing to recognize that some experts contribute far more significantly to the final output than others.
A Smarter Approach to Expert Management
Researchers from Nanjing University, Tsinghua University, and Honor Device Co.,Ltd have introduced an innovative solution called Importance-Driven Expert Scheduling. Unlike previous methods, this approach doesn’t treat all experts equally. It leverages the ‘importance’ of an expert, as indicated by its router’s gate score, to make smarter decisions about memory management and computation. The core idea is to prioritize high-importance experts and strategically substitute low-importance ones.
The system works by observing that in a fine-grained MoE framework, only a few active experts truly achieve high scores and significantly influence the output. Many others have low scores, similar to inactive experts. The new approach focuses on:
- Prefetching Top-Score Experts: Only the most critical, high-scoring active experts that reside in CPU memory are proactively loaded onto the GPU. This ‘pre-loading’ happens in advance, overlapping with other computations to hide the data transfer time.
- Substituting Low-Score Experts: Instead of incurring slow PCIe transfers or CPU computations for low-score active experts, the system replaces them with functionally similar inactive experts that are already cached in GPU memory. This substitution has a negligible impact on accuracy because these experts contribute minimally to the final result.
This intelligent design significantly reduces memory usage and data transfer, largely eliminating the overhead associated with PCIe. It also introduces a scheduling policy that maximizes the reuse of experts already in GPU memory, further boosting efficiency.
How the System Works
The Importance-Driven Expert Scheduler integrates several key components:
- Expert-cache Router: This component classifies active experts into ‘top-score’ and ‘low-score’ categories. It then intelligently replaces low-score active experts (if they are not already in GPU memory) with inactive experts that are already GPU-resident and have similar scores.
- Online Prefetching Top-score Experts: This mechanism predicts which high-scoring experts will be needed next and preloads them onto the GPU. Crucially, this prediction process is performed on the GPU itself, minimizing its impact on overall performance.
- Cache Eviction: To manage GPU memory effectively, a score-based eviction policy is used. Experts with the lowest average activation scores over recent iterations are removed first, ensuring that important experts are retained.
- CPU-Load Balancer: For any remaining active experts in CPU memory, this balancer dynamically decides whether to transfer them to the GPU or compute them directly on the CPU. It aims to harmonize the loading time with CPU computation time to minimize idle periods.
Also Read:
- GPT-OSS-20B: Unpacking the Deployment Efficiency of OpenAI’s Open-Weight MoE Model
- Unlocking NPU Potential for Mobile LLMs with Dynamic Sparse Attention
Impressive Results
Extensive evaluations of this approach demonstrate significant performance improvements. The system delivers a remarkable 48% lower decoding latency, which means faster responses from the LLM. It also achieves an impressive over 60% expert cache hit rate, indicating much more efficient use of the limited GPU memory. All these benefits are achieved while maintaining nearly lossless accuracy, proving that the strategic substitutions do not compromise the model’s performance.
The research was tested across various MoE models, including deepseek-moe-16b, XVERSE-MoE-A4.2B, and Qwen2-57B-A14B, and on different hardware configurations, from NVIDIA RTX 3080 Ti to A6000 GPUs. It consistently outperformed existing baseline systems like llama.cpp, MoE-Infinity, DeepSpeed, and HybriMoE.
This innovative Importance-Driven Expert Scheduling method offers a robust and effective solution for deploying powerful LLMs on resource-constrained edge devices, paving the way for more responsive and private AI applications in our daily lives. You can read the full research paper here.
