TLDR: ButterflyQuant is a new method for quantizing Large Language Models (LLMs) to ultra-low bitwidths (like 2-bit) without significant performance loss. It addresses the challenge of ‘outliers’ in LLM activations, which typically hinder extreme quantization. Unlike previous methods that use fixed, non-adaptive rotations, ButterflyQuant employs ‘learnable butterfly transforms’ with continuous rotation angles. This allows the model to adapt its rotations to the unique outlier patterns of different LLM layers, ensuring optimal information distribution for quantization. The method is efficient, guarantees orthogonality, and achieves significantly lower perplexity and higher accuracy retention compared to state-of-the-art techniques, making LLMs more deployable on consumer hardware.
Large Language Models (LLMs) have transformed artificial intelligence, but their immense size often makes them difficult to deploy on standard consumer hardware. This is primarily due to their massive memory requirements. One common solution is quantization, a technique that reduces the numerical precision of the model’s weights and activations, effectively compressing it. While effective, pushing quantization to extreme levels, such as 2-bit precision, often leads to a significant drop in performance. This degradation is largely caused by ‘outliers’ – unusually large values in the model’s activations that dominate the numerical range and make it difficult to represent accurately with very few bits.
Traditional rotation-based quantization methods, like QuIP and QuaRot, attempt to solve this outlier problem by applying an orthogonal transformation before quantization. This transformation redistributes the activation values, smoothing out the outliers without changing the layer’s overall output. These methods often use fixed transforms, such as Hadamard matrices, which are mathematically elegant and achieve optimal worst-case coherence (a measure of how evenly information is distributed). However, a critical limitation of these fixed transforms is their inability to adapt to the unique characteristics of different layers within an LLM.
Research has shown that various transformer layers in an LLM exhibit distinct outlier patterns. For instance, attention layers might have positive-skewed outliers, while different Multi-Layer Perceptron (MLP) layers show outliers in negative regions or near distribution boundaries. A one-size-fits-all approach with a fixed rotation simply cannot optimally address these diverse challenges across the network.
Introducing ButterflyQuant: Adaptive Quantization for LLMs
A new research paper, “ButterflyQuant: LLM Quantization through Learnable Orthogonal Butterfly Transforms”, introduces a novel approach to overcome this limitation. Developed by Bingxin Xu, Zhen Dong, Oussama Elachqar, and Yuzhang Shang, ButterflyQuant replaces these fixed Hadamard rotations with ‘learnable butterfly transforms’. These transforms are structured orthogonal matrices that are parameterized by continuous Givens rotation angles. Unlike Hadamard matrices, which use discrete +1 or -1 entries and cannot be optimized with gradients, butterfly transforms use continuous angles, allowing them to be smoothly optimized using gradient-based learning.
This continuous parameterization is a game-changer. It enables ButterflyQuant to learn layer-specific rotations that are tailored to each layer’s unique outlier distribution. The method guarantees orthogonality by construction, which is crucial for maintaining the theoretical benefits of rotation-based quantization. Furthermore, it achieves an efficient computational complexity of O(n log n) with a remarkably small number of learnable parameters (n log n / 2), making it practical for large models.
How ButterflyQuant Works
Butterfly transforms factorize into a series of sparse layers, each applying independent 2×2 Givens rotations. These layers create a hierarchical, crossing pattern, much like the wings of a butterfly, which efficiently mixes information across dimensions. This structure allows them to represent Hadamard matrices exactly as a special case, but with the added flexibility of learnable angles.
To further enhance quantization, ButterflyQuant introduces a uniformity regularization on the post-transformation activations. This encourages a smoother distribution of values across quantization bins, which is ideal for low-bit compression. The learning process itself is incredibly lightweight, requiring only 128 calibration samples and converging in minutes on a single GPU – a negligible one-time cost compared to the benefits gained.
For LLMs with dimensions that are not powers of two (e.g., LLaMA-2-13B’s 5120 dimension), ButterflyQuant employs composite transforms based on Kronecker products. This allows it to combine smaller orthogonal transforms, using butterfly transforms for power-of-2 components and other minimal parameterizations like Cayley transforms for non-power-of-2 factors, all while maintaining orthogonality and efficiency.
Impressive Results in 2-bit Quantization
The experimental results are compelling. On LLaMA-2-7B with 2-bit weight quantization (W2A16), ButterflyQuant achieved a perplexity of 15.4, significantly outperforming QuaRot’s 22.1 and GPTQ’s 36.77. On reasoning tasks, ButterflyQuant retained an impressive 88% of the FP16 model’s accuracy on average, whereas other baselines typically retained only 65-73%. These consistent improvements across both LLaMA-2-7B and LLaMA-2-13B models validate the effectiveness of its layer-adaptive approach.
Ablation studies further confirmed the design choices: initializing the learnable transforms with an identity matrix (no rotation) proved more effective than Hadamard or random initializations, allowing for gradual, stable learning. The rapid convergence and the significant reduction in quantization error compared to fixed Hadamard transforms highlight the power of this learnable, adaptive strategy.
Also Read:
- Merge-of-Thought Distillation: Unifying AI Reasoning Abilities
- LM-Searcher: Unifying Neural Architecture Search Across Domains Using LLMs
The Future of LLM Compression
ButterflyQuant represents a significant step forward in extreme LLM quantization. By bridging classical signal processing with modern deep learning through learnable structured transforms, it offers a practical and robust solution for deploying large language models on resource-constrained hardware. This approach demonstrates that continuous parameterization of orthogonal transforms can fundamentally change what is achievable in ultra-low-bit compression, making powerful LLMs more accessible than ever before.
