TLDR: CoMoE is a new framework that combines expert aggregation and offloading to efficiently deploy large Mixture-of-Experts (MoE) language models on resource-constrained edge devices. It dynamically adapts to real-time resource conditions, reducing memory usage by 70% and inference latency by 10.5%, enabling large models like Switch-Base-128 (7.4B parameters) to run on mobile edge hardware.
Large Language Models, or LLMs, have become incredibly powerful tools, but their massive size often makes them difficult to use on smaller devices like smartphones or smart home gadgets. A popular solution to this challenge is the Mixture-of-Experts (MoE) architecture. MoE models are designed to be more efficient by only activating a small portion of their “experts” for each task, saving on computational costs while still offering vast capabilities.
However, bringing these advanced MoE models to resource-limited mobile edge computing environments comes with significant hurdles. Edge devices typically have far less memory and processing power than cloud servers. MoE models, despite their computational efficiency, still require storing all their expert parameters, leading to a large memory footprint. Additionally, the dynamic nature of selecting and loading different experts can introduce substantial communication delays, especially over limited network bandwidths at the edge.
Existing research has tried to tackle these issues through two main approaches: expert aggregation and expert offloading. Expert aggregation techniques aim to reduce the model’s size by combining or distilling experts. Expert offloading strategies manage memory by moving inactive experts to slower storage (like CPU memory) and bringing them back when needed. While these methods offer improvements, they often work in isolation, failing to adapt to the constantly changing resources of edge environments, and don’t fully consider how expert selection influences offloading performance.
To overcome these limitations, researchers have proposed a new framework called CoMoE: Collaborative Optimization of Expert Aggregation and Offloading for MoE-based LLMs at Edge. CoMoE is a dynamic, resource-aware system that intelligently combines both expert aggregation and offloading strategies. It constantly monitors the real-time status of device resources, network conditions, and even the characteristics of the input data to make smart decisions.
CoMoE’s approach involves several key innovations. First, it systematically analyzes and optimizes how experts are grouped and merged (aggregation granularity) and how they are stored and moved (offloading strategies). Unlike previous methods, CoMoE understands the deep connections between deciding which experts to activate and where to store them, leading to much better overall performance.
Second, CoMoE introduces a unique two-level adaptation mechanism. It can make broad adjustments, like selecting a pre-optimized version of the model with fewer experts, for longer-term resource changes. For rapid, short-term fluctuations, it uses fine-grained expert scheduling and prefetching. This dual approach helps it remain efficient even in highly volatile edge environments.
Third, CoMoE offers a specialized optimization strategy for models with “Encoder-Decoder” architectures. It recognizes that the Encoder part, which processes input, can benefit more from aggressive aggregation and offloading due to its parallel processing nature. The Decoder part, which generates output sequentially, requires minimal intervention to avoid performance bottlenecks.
Extensive experiments on real mobile edge testbeds have shown remarkable results. CoMoE achieved approximately a 70% reduction in memory usage compared to standard methods. It also delivered 10.5% lower inference latency than existing expert offloading techniques, all while maintaining the model’s performance stability. For large-scale MoE models, such as the 7.4-billion-parameter Switch-Base-128, CoMoE reduced memory requirements from a hefty 15.6GB to a manageable 4.7GB. This breakthrough enables the deployment of such powerful models on resource-constrained mobile edge devices that previously could only handle much smaller models.
Also Read:
- Boosting LLM Performance on Edge Devices: A Dual Approach to Efficient Deployment
- Optimizing LLM Deployment: A New Approach to Reduce Network Traffic in MoE Models
The collaborative optimization framework makes it possible to efficiently run large MoE models, which were once confined to data centers, directly on edge devices. This opens up exciting new possibilities for AI applications in mobile and Internet of Things (IoT) environments, democratizing access to advanced language models. You can read the full research paper here.
