TLDR: AdaptCache is a novel system that enhances Large Language Model (LLM) performance by intelligently managing KV caches across hierarchical storage (DRAM and SSD). It employs adaptive lossy compression, dynamically selecting the optimal compression algorithm, rate, and device placement for each KV cache entry. This strategy significantly boosts DRAM cache hit rates, reduces loading delays, and maintains high generation quality, resulting in faster ‘time to first token’ and improved output quality compared to conventional methods.
Large Language Models (LLMs) are becoming increasingly common in various applications, from chat assistants to complex agentic systems. A key challenge in serving these models efficiently is managing the ‘KV cache’ – a memory component that stores previously processed context to avoid redundant computations and speed up inference. However, as LLM applications scale, the sheer size of these KV caches can quickly exceed the capacity of even high-end GPU and CPU memory, necessitating the use of hierarchical storage like DRAM and SSD.
Traditional approaches to storing KV caches across DRAM (faster, smaller) and SSD (slower, larger) often face a significant hurdle: most cache hits end up coming from the slower SSD, leading to noticeable delays. This is because the limited space in faster memory forces a large portion of the KV caches to be offloaded to disk.
Introducing AdaptCache: A Smarter Way to Manage KV Caches
A new system called AdaptCache addresses this problem by introducing an intelligent, adaptive approach to lossy KV cache compression. Instead of applying a uniform compression strategy, AdaptCache dynamically decides the best compression algorithm, compression rate, and even the device placement (DRAM or SSD) for each individual KV cache entry. The goal is to maximize the number of cache hits on the faster DRAM while minimizing loading delays, all without significantly compromising the quality of the generated text.
The core idea is that not all KV cache data is equally important or equally compressible. Some contexts might be better suited for ‘token dropping’ (removing less critical tokens), while others might benefit more from ‘quantization’ (reducing the precision of numerical data). AdaptCache also considers how frequently a KV cache is reused and how easily it can be compressed when deciding where to store it and at what compression level.
How AdaptCache Works Under the Hood
AdaptCache operates with three main components:
- Estimator: This component profiles different compression methods and storage devices offline to understand their impact on quality and delay. It also estimates how often a KV cache entry might be reused in the future.
- Policy Optimizer: This is the brain of AdaptCache. It uses a ‘utility metric’ that balances quality and loading delay, combined with the estimated future usage frequency, to make real-time decisions. When new KV cache entries are generated or existing ones need to be managed, the policy optimizer greedily selects the compression choice (to evict, store in full quality, or compress) that offers the best ‘marginal utility gain’ – essentially, the most benefit for the space saved.
- Executor: This component carries out the compression and storage decisions made by the policy optimizer.
By adaptively adjusting compression and placement, AdaptCache can store many more KV cache entries in the faster CPU memory (DRAM) than static compression methods. This leads to a much higher hit rate on DRAM and, consequently, a significantly lower average loading time from disk, all while maintaining comparable or even better generation quality.
Also Read:
- Enhancing Diffusion LLM Performance with Adaptive Block Sizing
- Faster LLM Interactions Through Precomputed Storage
Impressive Performance Gains
Evaluations using the popular Llama-3.1-8B-Instruct model across various LongBench datasets (covering summarization, question answering, and coding tasks) showed remarkable improvements. Compared to existing fixed compression baselines, AdaptCache achieved 1.43 to 2.4 times faster ‘time to first token’ (TTFT) at the same quality level. Alternatively, it delivered 6% to 55% quality improvements at the same delay.
For instance, in a coding task, AdaptCache achieved DRAM cache hit rates as high as 81% (and even higher in some configurations), significantly outperforming KIVI LRU, which managed only 38% at a 2-bit quantization. This demonstrates AdaptCache’s ability to intelligently prioritize and compress KV caches from longer contexts or those with high information redundancy, ensuring overall high response quality and faster inference.
This innovative approach to KV cache management, detailed in the research paper AdaptCache: KV Cache Native Storage Hierarchy for Low-Delay and High-Quality Language Model Serving, marks a significant step forward in optimizing LLM serving systems for both speed and quality.
