TLDR: FOEM is a new post-training quantization (PTQ) method for large language models (LLMs) that improves accuracy by explicitly compensating for previously overlooked first-order errors in quantization loss. It achieves this by approximating gradients efficiently and using precomputed Cholesky factors, significantly outperforming existing methods like GPTQ while maintaining minimal computational overhead, making LLMs more efficient and deployable.
Large Language Models (LLMs) like Llama have become incredibly powerful, excelling in tasks from understanding language to generating code. However, their massive size and computational demands make them challenging to deploy, especially on devices with limited resources. This is where quantization comes in, a technique that compresses these models by converting their high-precision data into a lower-bit format.
Post-training quantization (PTQ) is a particularly efficient method because it doesn’t require additional training. A well-known PTQ method is GPTQ, which works by estimating and compensating for errors introduced during quantization. It uses a mathematical concept called a Taylor expansion, focusing on the second-order terms to correct these errors. The assumption has always been that the first-order errors are so small they can be ignored in models that are already well-trained.
Addressing Overlooked Errors in LLM Quantization
New research challenges this assumption. The paper “First-Order Error Matters: Accurate Compensation for Quantized Large Language Models” introduces a novel PTQ method called FOEM. The authors reveal that as the quantization process progresses and errors are compensated for step-by-step, small first-order deviations can accumulate. These deviations, initially thought to be negligible, actually become significant, making the traditional assumption flawed.
FOEM addresses this by explicitly including these first-order gradient terms in its error compensation process. Instead of using complex and costly backpropagation to calculate these gradients, FOEM approximates them by simply looking at the difference between the current “latent” weights (the weights being adjusted) and their original, full-precision values. This clever approximation keeps the computational overhead minimal.
Furthermore, FOEM makes the compensation process efficient by using precomputed Cholesky factors. These are mathematical components that help quickly recover the inverse of Hessian submatrices, which are crucial for accurate error compensation, without needing to recompute them from scratch every time.
Demonstrated Performance Gains
The researchers conducted extensive experiments across various Llama models and benchmarks. Their findings show that FOEM consistently outperforms the classical GPTQ method. For instance, when quantizing models to 3-bit weights, FOEM significantly reduces the perplexity (a measure of how well a language model predicts a sample of text) of Llama3-8B by 89.6%. For the larger Llama3-70B model, FOEM improved the 5-shot MMLU accuracy (a benchmark for multi-task language understanding) from GPTQ’s 51.7% to an impressive 74.9%, getting much closer to the full-precision model’s performance of 78.6%.
What’s more, FOEM is highly compatible and can be seamlessly integrated with other advanced quantization techniques like GPTAQ and SpinQuant. When combined, it yields even further improvements, especially in challenging scenarios where weights, activations, and KV cache are all quantized to 4 bits (W4A4KV4 setting). This further narrows the accuracy gap with full-precision models, surpassing what current state-of-the-art methods achieve.
Also Read:
- Quantizing Text Classifiers: How Calibration Data Shapes Performance on Edge Devices
- Compactor: A New Approach to Efficient LLM Memory Management
Efficiency and Future Impact
In terms of efficiency, FOEM introduces very little additional computational cost. For example, in weight-only quantization on Llama3-8B, our FOEM method only increased the quantization time by a mere 0.6% compared to GPTQ. This demonstrates that FOEM successfully balances enhanced numerical precision with practical deployment constraints.
This new approach represents a significant step forward in making LLMs more accessible and efficient for real-world applications. By accurately compensating for previously overlooked first-order errors, FOEM enables higher accuracy at lower bit-widths, paving the way for broader deployment of powerful language models on a wider range of hardware. You can read the full research paper for more details at this link.
