TLDR: This research extends lossless compression techniques, previously applied to higher-precision neural network components, to lower-precision formats like FP8 and FP4, and to key-value (K/V) caches in large language models. By separating and entropy-coding floating-point exponents, the method achieves significant size reductions (up to 83% for FP8) for weights and checkpoints. Crucially, it also demonstrates that K/V caches exhibit compressible patterns, offering real-time memory savings during LLM inference, addressing critical deployment challenges for growing AI models.
As deep learning models continue to grow in size and complexity, the challenge of storing and transmitting them efficiently becomes increasingly critical. While techniques like quantization and pruning can reduce model sizes, they often come with a trade-off in precision or structure. This is where lossless compression steps in, aiming to shrink models without altering their original numerical values.
Recent advancements, particularly the ZipNN framework, have shown promising results in lossless compression for neural networks. ZipNN identified that the exponent part of floating-point numbers, especially in formats like BF16, has a highly skewed distribution, making it ideal for compression using methods like Huffman coding. However, ZipNN primarily focused on higher-precision formats and did not address the significant memory footprint of key-value (K/V) caches in large language models (LLMs).
Extending Compression to Low-Precision Formats
A new research paper, titled “Lossless Compression of Neural Network Components: Weights, Checkpoints, and K/V Caches in Low-Precision Formats” by Anat Heilper and Doron Singer from Intel, extends these lossless compression techniques to the increasingly popular lower-precision floating-point formats: FP8 and FP4. These formats are gaining traction for their efficiency in inference, but it was previously unclear if their limited bit-widths would still allow for effective compression.
The core methodology involves separating the exponent and mantissa components of floating-point numbers and compressing them independently using entropy coding, such as Huffman coding. The researchers found that even in these ultra-low-precision formats, the exponent distributions remain skewed enough to yield significant compression. For instance, their evaluation showed compression ratios up to 62% for BF16 and an impressive 83% for FP8, meaning the compressed data was only 17% of its original size.
Compressing Key-Value Caches in LLMs
A significant contribution of this work is the investigation into the compressibility of key-value (K/V) cache tensors, which are crucial components in large language models (LLMs) during inference. These caches can consume substantial memory, growing linearly with sequence length and batch size, often becoming a bottleneck for deployment. The research demonstrates that K/V caches also exhibit compressible patterns, particularly in their exponent components, enabling memory savings during deployment without altering the model’s behavior.
For FP8 K/V caches, exponent compression ratios ranged from 0.25 to 0.45, while BF16 caches often saw ratios below 0.20. This is a novel finding, as prior work had not explored lossless compression for K/V caches. The ability to compress these caches in real-time offers a complementary solution to existing memory management strategies, potentially allowing for longer sequences or higher batch sizes in memory-constrained environments.
Insights into FP4 Compression
When it comes to FP4, the most aggressive low-precision format, the researchers found that the 4-bit quantized values themselves do not exhibit consistent statistical redundancy and are not suitable for lossless compression. Their bit patterns appeared close to uniformly random. However, the associated scaling factors used in FP4 blocks do show structure and can be effectively compressed using Huffman coding, offering measurable gains.
Also Read:
- Accelerating Large Language Models with Arbitrary Precision Computing
- New Research Uncovers How Quantization Affects Different Types of Knowledge in Large Language Models
Implications for Model Deployment
The findings of this research highlight that lossless compression remains a valuable tool for optimizing neural network deployment, even as models shift towards lower-precision formats. By targeting specific components like exponents and scaling factors, significant reductions in storage, transmission, and runtime memory can be achieved. While real-time compression of K/V caches presents performance challenges, the potential memory savings are substantial and could lead to more efficient and accessible long-sequence inference for LLMs.
This work paves the way for further optimizations, including hardware acceleration for compression and integration into inference runtimes, ensuring that the benefits of lossless compression can be fully realized in practical applications. You can read the full research paper here.
