TLDR: MC# is a new framework that significantly compresses large Mixture-of-Experts (MoE) language and vision models. It uses a two-stage process: Pre-Loading Mixed-Precision Quantization (PMQ) to reduce model size by assigning optimal bit-widths to experts, and Online Top-any Pruning (OTP) to dynamically select and activate only essential experts during inference. This results in substantial memory and computational savings with minimal performance loss, making powerful MoE models deployable on more accessible hardware.
Large language models (LLMs) and vision-language models (VLMs) have seen incredible advancements, often leveraging a sophisticated architecture known as Mixture-of-Experts (MoE). MoE models enhance capabilities by using multiple specialized ‘expert’ networks, activating only a subset for each input. This design boosts model capacity while maintaining efficiency through sparse activation. However, despite this sparsity, these models still demand substantial memory and computational power because all experts must be preloaded into memory, and several are activated for each input. This creates a significant hurdle for deploying these powerful AI models on hardware with limited resources.
Introducing MC#: A Unified Compression Framework
To tackle these challenges, a new framework called MC# (Mixture-Compressor-sharp) has been proposed. Developed by a team of researchers including Wei Huang, Yue Liao, Yukang Chen, Jianhui Liu, Haoru Tan, Si Liu, Shiming Zhang, Shuicheng Yan, and Xiaojuan Qi, MC# offers a unified approach to aggressively compress MoE-LLMs and MoE-VLMs. It achieves this by combining two key strategies: static quantization and dynamic expert pruning, both of which leverage the importance of individual experts and input tokens.
Pre-Loading Mixed-Precision Quantization (PMQ)
The first component of MC# is Pre-Loading Mixed-Precision Quantization (PMQ). This method focuses on reducing the storage and loading overhead of MoE models. PMQ intelligently assigns different bit-widths (e.g., 1, 2, or 3 bits) to various experts. Instead of uniformly quantizing all experts to the same low bit-width, which can lead to significant performance drops, PMQ formulates this adaptive bit allocation as a linear programming problem. It considers factors like how frequently an expert is activated, its activation weight, and the potential quantization error. This ensures that more critical experts retain higher precision, while less important ones can be more aggressively compressed, striking an optimal balance between model size and performance. Other parts of the model, like attention mechanisms and shared experts, are uniformly quantized to 4 bits, as their contribution to the overall model size is minimal.
Online Top-any Pruning (OTP)
The second crucial part of MC# is Online Top-any Pruning (OTP), designed to reduce runtime computation. During inference, OTP dynamically selects only a subset of experts for each input token. Unlike traditional rule-based pruning methods, which can be inflexible, OTP models expert activation as a learnable distribution using a technique called Gumbel-Softmax sampling. This makes the pruning process differentiable, allowing the model to learn the most effective expert selection strategy for each token. By dynamically pruning redundant experts, OTP further enhances inference efficiency, especially in ultra-low-bit MoE models, without significantly compromising performance.
Also Read:
- Dynamic Nested Depth: A New Approach to Smarter Language Models
- Enhancing LLM Efficiency with Fine-grained Low-Rank Compression
Remarkable Results and Future Implications
The effectiveness of MC# has been demonstrated through extensive experiments. For instance, on the DeepSeek-VL2 model, MC# achieved an impressive 6.2 times reduction in weight, averaging 2.57 bits per parameter, with only a 1.7% drop in accuracy across five multimodal benchmarks compared to the original 16-bit model. Furthermore, OTP alone reduced expert activation by an additional 20% with less than a 1% performance loss. These results highlight that MC# not only significantly compresses model size and reduces computational costs but also maintains high performance, even outperforming some uncompressed smaller models. This dual compression strategy makes powerful MoE-based models more accessible, enabling their deployment on consumer-grade hardware and edge devices, moving them beyond multi-GPU cluster environments. For more in-depth information, you can refer to the full research paper.
