TLDR: This research introduces a collaborative compression framework combining expert pruning, mixed-precision quantization, and activation optimization to deploy ultra-large Mixture of Experts (MoE) models on resource-constrained edge platforms. The framework successfully reduced the DeepSeek-V3 model from 1.3TB to 103GB, enabling its deployment on a 128GB memory laptop while maintaining high accuracy, outperforming traditional uniform low-bit quantization methods.
Large Language Models (LLMs) are becoming incredibly powerful, and a key architecture enabling their growth is the Mixture of Experts (MoE). MoE models can significantly increase a model’s capacity without a proportional increase in computational cost. However, these ultra-large MoE models, often containing hundreds of billions of parameters, demand massive amounts of memory and storage, making them incredibly difficult to deploy on everyday devices like laptops or smartphones, which have limited resources.
Traditional methods to shrink these models, such as pruning (removing unnecessary parts) or quantization (reducing the precision of data), often fall short when faced with the extreme compression ratios needed for edge deployment. Applying these techniques too aggressively can severely degrade the model’s accuracy and the quality of its output, sometimes rendering it unusable.
To overcome this significant challenge, researchers have introduced a novel collaborative compression framework. This framework combines three powerful strategies: expert pruning, mixed-precision quantization, and activation optimization. By working together, these methods can achieve a much higher compression ratio while still maintaining excellent performance and output quality.
The framework begins with
expert pruning
, which identifies and removes less important “experts” within the MoE model. This step significantly reduces the overall parameter count. Following pruning,
hardware-aware activation adjustment
fine-tunes how experts are activated, ensuring that the model’s operation aligns with the reduced expert pool and the specific memory and computational limits of the target device. Finally, Also Read:
- New Scaling Laws for Combining Large Language Models
- SpecExit: Smarter, Faster Reasoning for Large Language Models
mixed-precision quantization
is applied. Instead of uniformly reducing all data to a very low precision (which can harm accuracy), this method intelligently assigns different precision levels to different parts of the model based on their sensitivity. More critical parts retain higher precision, while less sensitive parts are compressed more aggressively, all while staying within a defined memory budget.
A remarkable achievement of this framework is its ability to shrink the ultra-large DeepSeek-V3 MoE model, which originally required 1.3 terabytes (TB) of storage, down to just 103 gigabytes (GB). This compressed model can then be successfully deployed on platforms with strict memory limits, such as a laptop with a total of 128GB memory. This was previously considered an impossible feat with existing methods.
Extensive experiments have shown that this collaborative compression approach not only results in smaller model sizes but also achieves higher accuracy across various benchmarks compared to traditional uniform low-bit quantization methods. For instance, the 103GB compressed DeepSeek-V3 model outperformed a 140GB model using uniform quantization in several reasoning tasks, demonstrating its practical effectiveness.
The success of this framework marks a significant step towards making powerful, large-scale AI models accessible on resource-constrained edge devices, opening up new possibilities for on-device AI applications. For more technical details, you can refer to the full research paper available here.
