TLDR: LRQK (Low Rank Query and Key attention) is a novel two-stage framework designed to make Large Language Models (LLMs) more memory-efficient for long-context inference. It achieves this by jointly decomposing query and key matrices into compact low-rank factors and employing a mixed GPU-CPU cache with a ‘hit-and-miss’ mechanism. This approach selectively transfers only necessary full-precision key-value pairs, significantly reducing GPU memory costs and CPU-GPU data movement while preserving exact attention outputs and maintaining high accuracy, outperforming existing sparse attention methods.
Large Language Models (LLMs) have transformed how we interact with AI, enabling advanced capabilities from document understanding to generating long-form text. However, as these models process increasingly longer inputs, a significant challenge arises: the prohibitive GPU memory cost associated with the Key-Value (KV) cache. This cache stores historical information to prevent redundant computations, but its size grows linearly with the input sequence length, quickly becoming a bottleneck for devices with limited resources.
Existing solutions, such as quantizing KV pairs (reducing numerical precision) or pruning them (selectively removing less important ones), often come with trade-offs. Quantization can lead to a loss in numerical precision, while pruning risks discarding crucial information that might become important later in the context. Offloading the entire KV cache to CPU memory, while saving GPU space, introduces substantial latency due to frequent data transfers between the CPU and GPU.
Introducing Low Rank Query and Key Attention (LRQK)
A new approach, called Low Rank Query and Key attention (LRQK), offers an innovative solution to this memory bottleneck. LRQK is a two-stage framework designed to enable efficient long-context inference in LLMs without sacrificing accuracy.
The first stage, known as the ‘prefill’ stage, involves jointly decomposing the full-precision query and key matrices into more compact, lower-dimensional factors. This process effectively reduces the complexity and memory footprint of these critical components.
In the subsequent ‘decode’ stage, these low-dimensional projections are used to compute ‘proxy attention scores’. This allows for a much faster estimation of attention, operating in a time complexity that scales with the sequence length and the chosen low rank (O(lr)).
Smart Cache Management for Precision and Efficiency
One of LRQK’s most significant innovations is its mixed GPU-CPU cache management system. Instead of constantly moving large amounts of data, LRQK employs a ‘hit-and-miss’ mechanism. It intelligently selects only the ‘top-k’ most relevant tokens and a small, fixed set of ‘recent tokens’ to keep in the faster GPU cache. If a required KV pair is not in the GPU cache (a ‘miss’), only that specific missing full-precision KV pair is fetched from the slower, larger CPU cache. This selective transfer drastically reduces CPU-GPU data movement, preserving the exact attention outputs and maintaining the model’s original performance.
Also Read:
- FALQON: Speeding Up LLM Fine-tuning with Merged Low-Bit Adapters
- GradLite: A New Optimizer for Memory-Efficient LLM Training
Key Advantages and Performance
The LRQK method brings several advantages:
- Joint Low Rank Approximation: It optimizes low rank approximations for both query and key matrices, reducing computational complexity while maintaining high representational accuracy.
- Precision-Preserving Attention: Crucially, the final attention layer operations utilize the original, full-precision query, key, and value vectors. The low rank approximations are only used as proxies for efficient selection, ensuring mathematical fidelity and model performance.
- Mixed Cache Management: The hybrid GPU-CPU storage system, with its active token retention and hit/miss buffer, minimizes cross-device data transfer and prioritizes recently accessed tokens, which are empirically shown to have high attention scores.
Extensive experiments conducted on benchmarks like RULER and LongBench, using models such as LLaMA-3-8B and Qwen2.5-7B, demonstrate that LRQK performs comparably to or even surpasses leading sparse-attention methods in long-context scenarios. It achieves substantial memory savings, allowing LLMs to process contexts that would otherwise cause out-of-memory errors, all with minimal loss in accuracy. For those interested in exploring the implementation, the code is available on GitHub.
While LRQK significantly reduces data transfer overhead, the authors note that CPU-side indexing operations currently pose the primary performance bottleneck, an area for future improvement. Additionally, like many advanced methods, LRQK’s hyperparameters may require some task-specific tuning for optimal performance.
This research represents a significant step towards making powerful long-context LLMs more accessible and efficient, especially on resource-constrained devices. You can read the full paper here: Efficient Low Rank Attention for Long-Context Inference in Large Language Models.
