TLDR: Amber Pruner is a novel, training-free method that applies N:M activation sparsity to Large Language Models (LLMs) during the prefill stage, significantly accelerating linear computations (over 55%) with less than 1% accuracy loss. It uses a Robust-Norm Scoring mechanism and a Layer Skipping Strategy to maintain performance. When combined with W8A8 quantization in ‘Outstanding-sparse,’ it further enhances efficiency, demonstrating a practical path for deploying faster and more compact LLMs without retraining.
Large Language Models (LLMs) have become incredibly powerful, but their size often makes them slow and expensive to run. To make them more efficient, researchers are constantly looking for ways to compress these models without losing too much of their performance. One promising technique is N:M sparsity, a method that allows only a specific number of non-zero elements (N) within every block of M elements, which is well-suited for modern hardware.
Traditionally, much of the focus in model compression has been on ‘weight sparsity,’ where the connections (weights) within the neural network are pruned. However, this often leads to a noticeable drop in accuracy, making it less practical for real-world applications. Another area, ‘activation sparsity,’ which involves making the intermediate calculations (activations) sparse, holds great promise for speeding up LLMs, especially during the ‘prefill’ stage where the model processes the initial input. The challenge with activation sparsity has been its reliance on extensive training and difficulties in generalizing across different models.
Introducing Amber Pruner: A Training-Free Approach
To overcome these limitations, researchers at Huawei Technologies have introduced a novel method called Amber Pruner. This innovative algorithm is designed to apply N:M activation sparsity without requiring any additional training or fine-tuning of the LLM. It specifically targets the ‘prefill’ stage of LLM inference, focusing on accelerating the linear projection layers, which are computationally intensive.
Amber Pruner works by intelligently identifying and retaining only the most important activation elements. Instead of a simple ‘top-k’ selection, it uses a sophisticated ‘Robust-Norm Scoring’ mechanism. This mechanism considers the importance of activations based on how much they contribute to the model’s output, ensuring that critical information is preserved. It also includes a ‘Layer Skipping Strategy,’ which is crucial for maintaining accuracy. This strategy identifies and bypasses certain sensitive layers that are more prone to performance degradation if heavily pruned, such as the ‘o_proj’ and ‘up_proj’ modules. Conversely, less sensitive layers like ‘down_proj’ are consistently targeted for pruning.
Through extensive experiments, Amber Pruner has shown impressive results. It can effectively sparsify and accelerate over 55% of linear computations in various LLMs, including LLaMA3.1-8B-Instruct, Qwen2-7B-Instruct, and even sparse Mixture-of-Experts (MoE) models like Qwen3-30B-A3B. What’s remarkable is that it achieves this with very little accuracy loss—typically less than 1% on common benchmarks, especially with an 8:16 sparsity ratio (meaning 8 non-zero elements in every 16). This minimal impact on performance makes it highly practical for deployment.
Also Read:
- Optimizing Large Language Model Efficiency with LeanK’s Smart Cache Pruning
- Optimizing LLM Performance with Intelligent KV Cache Compression
Outstanding-sparse: Combining Sparsity with Quantization
Further enhancing the efficiency, the researchers also proposed ‘Outstanding-sparse,’ a unified framework that integrates Amber Pruner with W8A8 quantization. Quantization is another compression technique that reduces the precision of numbers used in the model (e.g., from 32-bit to 8-bit), thereby saving memory and speeding up calculations. Outstanding-sparse intelligently adjusts the quantization process to work synergistically with activation sparsity, even expanding the activation range to make sparsity patterns more apparent.
The combination of Amber Pruner and W8A8 quantization in Outstanding-sparse yields competitive results, demonstrating that these two compression techniques can effectively complement each other. While some accuracy degradation is observed, the research indicates that most of it comes from the sparsity rather than the quantization itself, highlighting the effectiveness of their combined approach. This method also proved robust on MoE models, showing its broad applicability.
This work represents a significant step forward in making LLMs more efficient for real-world applications. By pioneering a training-free N:M activation sparsity method, Amber Pruner offers a practical solution to accelerate LLM inference, particularly during the prefill stage. Although current hardware still has limitations in fully supporting such fine-grained sparsity, this research provides foundational insights that are poised to guide the co-evolution of algorithms and architectures in the design of next-generation AI systems. For more details, you can refer to the full research paper: Amber Pruner: Leveraging N:M Activation Sparsity for Efficient Prefill in Large Language Models.
