TLDR: MoEpic is a new system designed to accelerate Mixture-of-Experts (MoE) Large Language Model (LLM) inference while significantly reducing GPU memory (VRAM) requirements. It achieves this by splitting AI experts into ‘top’ and ‘bottom’ segments, caching the ‘top’ parts of frequently used experts in VRAM, and dynamically prefetching ‘bottom’ parts. MoEpic also features an adaptive algorithm for VRAM budget allocation and expert split ratio determination across different layers, along with a priority-based cache policy. Experiments show MoEpic can cut GPU costs by half and reduce inference latency by 37.51% to 65.73% compared to other methods.
Large Language Models (LLMs) have transformed artificial intelligence, but their immense size, especially those built with a Mixture-of-Experts (MoE) architecture, demands significant GPU memory (VRAM). This high cost makes it challenging to deploy these powerful AI models locally, particularly in sensitive sectors like healthcare or finance where data privacy is paramount.
Current solutions often involve offloading expert parameters to CPU RAM to save VRAM. However, this comes with a major drawback: a substantial drop in inference speed due to slow data transfer between CPU and GPU. Existing methods try to mitigate this through caching frequently used experts or prefetching experts for the next computational step. Unfortunately, caching often suffers from a low ‘cache hit rate’ because VRAM is limited, meaning many experts still need to be loaded on demand. Prefetching, while helpful, often finds that the time it takes to load an expert is longer than the time the GPU spends computing, leading to unavoidable delays.
A new research paper, Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism, introduces an innovative system called MoEpic. Developed by Jiaming Yan, Jianchun Liu, Hongli Xu, and Liusheng Huang, MoEpic aims to overcome these limitations by intelligently managing how experts are stored and accessed.
The core idea behind MoEpic is a novel ‘expert split mechanism’. Instead of treating each expert as a single unit, MoEpic vertically divides each expert into two segments: a ‘top’ segment and a ‘bottom’ segment. The system then strategically caches the ‘top’ segments of frequently activated, or ‘hot’, experts in the limited VRAM. This approach allows more experts to have at least a part of themselves in VRAM, significantly improving the cache hit rate.
During inference, MoEpic predicts which experts will be needed for the next layer and prefetches their ‘bottom’ segments from CPU RAM. Since the ‘top’ segments of cached experts are already in VRAM, they don’t need to be fetched, which reduces the overall loading time. This reduction in loading time is crucial because it allows for more effective overlap between data transfer and GPU computation, making the entire process much faster.
MoEpic also tackles two critical challenges in optimizing MoE inference: how to allocate VRAM budget across different layers and how to determine the optimal split ratio for each expert. The paper proposes a sophisticated divide-and-conquer algorithm based on fixed-point iteration to adaptively configure the cache, ensuring that each layer receives an appropriate VRAM budget and expert split ratio. Furthermore, MoEpic introduces a priority-based cache policy, called LCP, which combines the strengths of both Least Recently Used (LRU) and Least Frequently Used (LFU) policies to manage the cache more effectively.
Extensive experiments conducted on popular MoE LLMs like Qwen1.5-MoE and Mixtral-8x7B demonstrate MoEpic’s superior performance. The system can save approximately half of the GPU cost while reducing inference latency by a remarkable 37.51% to 65.73% compared to existing baselines. This means faster responses from AI models and lower hardware expenses for deployment.
Also Read:
- Boosting MoE-LLM Performance and Speed with Smarter Expert Routing
- VoltanaLLM: Optimizing Energy Use for Large Language Model Serving
In essence, MoEpic represents a significant step forward in making powerful MoE LLMs more accessible and efficient, especially for on-premises deployments where VRAM is a bottleneck. By intelligently splitting experts and adaptively managing memory, MoEpic allows these advanced AI models to run faster and with less demanding hardware, paving the way for broader adoption.
