TLDR: CoSpaDi (Compression via Sparse Dictionary Learning) is a novel, training-free framework for compressing large language models (LLMs). Unlike traditional low-rank methods that use a single shared subspace, CoSpaDi employs a dense dictionary and a column-sparse coefficient matrix, enabling a more flexible “union-of-subspaces” representation. It leverages a small calibration dataset to optimize compression by minimizing functional reconstruction error, ensuring better model fidelity without fine-tuning. Experiments show CoSpaDi consistently outperforms state-of-the-art low-rank methods in accuracy and perplexity across various Llama and Qwen models, making LLM deployment more efficient.
Large Language Models (LLMs) have revolutionized many applications, from conversational AI to complex reasoning. However, their immense size and computational demands pose significant challenges for deployment, especially in environments with limited resources. This often necessitates compression techniques to make them more manageable.
Traditionally, one popular method for compressing LLMs after they’ve been trained is called low-rank weight approximation. This technique simplifies the complex weight matrices within an LLM by representing each column in a shared, lower-dimensional space. While efficient, this approach can be quite rigid, often leading to a noticeable drop in the model’s accuracy because it forces all parts of the model to conform to a single, simplified structure.
A new research paper introduces an innovative solution called CoSpaDi, which stands for Compression via Sparse Dictionary Learning. This framework offers a fresh perspective on LLM compression, moving beyond the limitations of conventional low-rank methods. CoSpaDi proposes a more flexible approach by replacing low-rank decomposition with a structured sparse factorization.
How CoSpaDi Works
Instead of relying on a single, shared low-dimensional space, CoSpaDi represents each weight matrix using a dense dictionary and a column-sparse coefficient matrix. Think of it like this: imagine you have a vast library of words (the dictionary atoms). Instead of trying to describe every book in the library using only a few common themes (low-rank), CoSpaDi allows you to pick a small, specific set of words from your library to describe each book accurately. This means different books (or different columns of the weight matrix) can be described using different, adaptively selected sets of words, offering much greater expressiveness.
This unique formulation enables what the researchers call a “union-of-subspaces” representation. It means that different parts of the original weight matrix can be approximated in distinct, specialized subspaces, providing more flexibility than a single, invariant basis. This adaptability is crucial for maintaining model fidelity.
Crucially, CoSpaDi is a “data-aware” strategy. It uses a small calibration dataset to optimize the compression process. Instead of just trying to make the compressed weights look like the original weights, CoSpaDi focuses on ensuring that the output activations of the compressed layers closely match those of the original layers. This minimizes the functional reconstruction error, meaning the model behaves very similarly to the uncompressed version, even without any fine-tuning after compression.
Also Read:
- Boosting Transformer Efficiency with Compressed Convolutional Attention
- PatternKV: A New Approach to Optimize LLM Memory and Speed
Key Advantages and Results
The benefits of CoSpaDi are significant. It preserves model fidelity better than state-of-the-art data-aware low-rank methods, especially at reasonable compression ratios (20-50%). The resulting structured sparsity also allows for efficient sparse-dense matrix multiplication, which can lead to faster inference times. Furthermore, CoSpaDi is compatible with post-training quantization, offering potential for even greater memory and latency gains.
The researchers evaluated CoSpaDi across multiple Llama and Qwen models, in both per-layer (compressing each layer independently) and per-group (sharing a dictionary across similar layers) settings. The results consistently demonstrated CoSpaDi’s superiority over existing data-aware low-rank methods in terms of both accuracy and perplexity (a measure of how well a language model predicts a sample of text). For instance, on Llama3 8B, CoSpaDi significantly outperformed SVD-LLM across various benchmarks and compression levels, maintaining higher average accuracy and lower perplexity.
This work, authored by Dmitriy Shopkhoev, Denis Makhov, Magauiya Zhussip, Ammar Ali, and Stamatios Lefkimmiatis, establishes structured sparse dictionary learning as a powerful and promising alternative to conventional low-rank approaches for efficient LLM deployment. You can read the full research paper for more details: COSPADI: COMPRESSING LLMS VIA CALIBRATION-GUIDED SPARSE DICTIONARY LEARNING.
