TLDR: SnapStream is a novel KV cache compression method for Large Language Models (LLMs) that combines SnapKV and StreamingLLM techniques. It addresses memory pressure and deployment challenges in industrial settings using static graphs and continuous batching. Implemented on SambaNova SN40L accelerators, SnapStream achieves 4x memory improvement and 4.3x decoding throughput increase with minimal accuracy loss for models like DeepSeek-R1, enabling efficient long-sequence decoding.
Large Language Models (LLMs) are becoming increasingly powerful, with many now boasting over 100 billion parameters and the ability to understand contexts exceeding 100,000 tokens. While impressive, this growth comes with a significant challenge: memory. These massive models require vast amounts of on-chip memory, especially for their “KV caches,” which store past information to help generate new text efficiently.
Existing techniques like StreamingLLM and SnapKV have shown promise in managing KV cache size while maintaining accuracy. However, their adoption in industrial settings, particularly with frameworks that use static graphs and continuous batching, has been slow. This is mainly due to the difficulty of integrating these modifications into standard attention algorithms and a lack of clear understanding regarding their accuracy implications on modern reasoning models.
A new research paper, titled “SNAPSTREAM: EFFICIENT LONG SEQUENCE DECODING ON DATAFLOW ACCELERATORS,” introduces a novel solution called SnapStream. Developed by Jonathan Li and a team of researchers, SnapStream is a KV cache compression method designed for large-scale deployment. The paper explores the accuracy of such techniques on models like Llama-3.1-8B-Instruct and DeepSeek-R1, demonstrating how SnapStream can be effectively deployed.
SnapStream addresses several practical considerations for modern LLM deployments:
Long Sequence Decoding
Many KV cache compression methods are evaluated on benchmarks with long inputs, compressing the cache once at the start. SnapStream, however, is designed to handle reasoning models that generate thousands of tokens, where the effects of continuous compression are crucial.
Continuous Batching
Cloud LLM deployments often use continuous batching to manage prefill (processing the initial prompt) and decoding (generating subsequent tokens) stages. SnapStream seamlessly integrates into this workflow, handling compression at different times for various batch elements.
Also Read:
- Boosting LLM Performance: How Processing-Near-Memory Redefines KV-Cache Management
- NVIDIA’s kvtc Breakthrough: Compressing LLM KV Caches for Enhanced Efficiency
Static Tensor Shapes
Production LLM systems often rely on fixed tensor shapes for efficient memory allocation. Unlike many existing compression methods that use dynamic shapes, SnapStream is implemented with fixed tensor sizes, avoiding fragmentation and improving efficiency.
At its core, SnapStream combines two techniques: SnapKV compression during the prefill phase and StreamingLLM during the decoding phase. This allows it to generate long sequences with a significantly smaller, fixed KV cache size. During prefill, it identifies and compresses less important tokens using SnapKV. Then, during decoding, it uses a StreamingLLM-inspired rolling window, effectively a “ring buffer,” to manage the most recent tokens while keeping the compressed and “sink” (initial important) tokens intact.
The researchers demonstrated SnapStream’s effectiveness in a real production setting, using a 16-way tensor-parallel deployment of DeepSeek-671B on SambaNova SN40L accelerators. This setup achieved 128k context length and up to 1832 tokens per second. The results were impressive: SnapStream enabled a 4x improvement in on-chip memory usage and introduced only minimal accuracy degradation on benchmarks like LongBench-v2, AIME24, and LiveCodeBench. This marks the first known implementation of sparse KV attention techniques deployed in a production inference system with static graphs and continuous batching.
The paper details the implementation on SN40L accelerators, highlighting how kernel fusion and tensor sharding optimize performance for both prefill and decoding stages. The additional compression logic introduced a modest latency overhead of only 2-5% during prefill, which is a small price to pay for the significant memory and throughput gains during decoding.
Specifically, SnapStream led to a 4x increase in the maximum attainable batch size and a substantial 4.3x improvement in decoding throughput for DeepSeek-R1-0528. This means more users can be served simultaneously, and responses are generated faster, all while using less memory. For more technical details, you can read the full paper here.
In conclusion, SnapStream offers a practical and efficient solution to the growing memory demands of large language models with long context lengths. By intelligently compressing KV caches and integrating seamlessly into existing production frameworks, it paves the way for more scalable and performant LLM deployments.
