TLDR: Helix Parallelism is a novel hybrid execution strategy that optimizes real-time decoding for Large Language Models (LLMs) with multi-million-token contexts. It addresses key bottlenecks by decoupling and reconfiguring GPU parallelism for attention and Feed-Forward Network (FFN) computations. By sharding KV caches and FFN weights across all available GPUs and employing a batch-wise overlap optimization (HOP-B), Helix significantly reduces Token-to-Token Latency (TTL) by up to 1.5x and enables up to 32x larger batch sizes, making ultra-long-sequence LLM inference practical and efficient.
Large Language Models (LLMs) are becoming increasingly powerful, capable of understanding and generating text over millions of tokens. This ability to handle vast amounts of information, known as ultra-long histories or contexts, is crucial for applications like AI assistants and copilots that need to maintain narrative coherence and support complex reasoning. However, delivering real-time responses, measured by Token-to-Token Latency (TTL), under these conditions presents significant challenges.
The primary hurdles in decoding multi-million-token LLMs are two-fold. Firstly, accessing the Key-Value (KV) cache during self-attention becomes incredibly expensive. The KV cache stores intermediate representations of previous tokens, and its size grows linearly with both context length and the number of concurrent requests (batch size). This rapidly overwhelms a GPU’s memory capacity and bandwidth, forcing systems to reduce batch sizes, which still leaves read times high and pushes latency beyond acceptable limits.
Secondly, reading Feed-Forward Network (FFN) weights also contributes heavily to latency. Generating each new token requires loading large FFN weights from memory. With small batch sizes, this cost cannot be spread out effectively, making FFN weight reads a dominant factor in the overall decoding time.
Traditional parallelism techniques, like Tensor Parallelism (TP), help by sharding FFN weights and attention heads across GPUs. While effective for FFNs, TP doesn’t scale well for attention when the TP width exceeds the number of KV heads. This leads to inefficient duplication of the KV cache on each GPU, limiting further parallelism and constraining the batch size. Another approach, KV Parallelism (KVP), like the Medha system, shards the KV cache across many GPUs, significantly reducing per-GPU memory usage. However, these methods often gather attention outputs onto a smaller, fixed group of TP GPUs for FFN computations, meaning the additional GPUs used for KVP are not fully utilized to accelerate FFNs, leaving FFN weight loads as a bottleneck.
Introducing Helix Parallelism
To address these fundamental bottlenecks, researchers from NVIDIA Corporation have introduced Helix Parallelism. This innovative hybrid execution strategy rethinks how sharding is applied, specifically by decoupling the mapping of attention and FFN computations in a temporal pipeline. The core idea is to reuse the same set of GPUs for both attention and FFN calculations, but with different, optimized parallelism strategies for each phase.
In the attention phase, Helix applies KV Parallelism (KVP) to shard the KV cache along the sequence dimension across a pool of GPUs. This eliminates full-cache replication, drastically cutting down memory footprint and bandwidth demands. To ensure exact attention behavior, a lightweight communication step is included where GPUs exchange partial attention outputs. This setup also combines with a limited form of Tensor Parallelism (TPA) across KV heads, ensuring no KV cache duplication occurs.
Immediately following the attention phase, the FFN phase reconfigures the *same* GPUs. For dense LLMs, all GPUs are used for Tensor Parallelism (TPF) to shard FFN weight matrices, accelerating weight reads. For Mixture-of-Experts (MoE) models, the GPUs are repartitioned into a TP × Expert Parallelism (EP) grid. This dynamic reconfiguration allows for much wider TP for FFNs than would be possible with traditional methods, without reintroducing the KV cache duplication problem during attention.
A key optimization within Helix is Helix HOP-B (Helix Overlap Pipeline – Batch-wise). This is a fine-grained pipelining strategy that cleverly overlaps the necessary communication steps with ongoing attention computation across the batch dimension. By doing so, HOP-B effectively hides communication latency, maintaining low Token-to-Token Latency (TTL) and preserving real-time responsiveness.
Also Read:
- any4: A New Approach to 4-bit Quantization for Large Language Models
- Power Attention: A New Approach to Efficient Long-Context AI Models
Performance and Impact
Evaluations conducted on NVIDIA’s latest GB200 NVL72 hardware, simulating million-token context lengths, demonstrate significant improvements. Compared to conventional parallelism approaches, Helix Parallelism:
- Reduces Token-to-Token Latency (TTL) by up to 1.5x at fixed batch sizes.
- Supports up to 32 times larger batches under the same latency budget for models like DeepSeek-R1.
- For Llama-405B, it yields a 1.13x improvement in maximum achievable interactivity and a 4x higher throughput and batch capacity.
These gains are attributed to Helix’s ability to shard both KV caches and FFN weights across all available devices, which reduces memory pressure and significantly increases compute efficiency. The HOP-B optimization is particularly critical for models where communication forms a larger fraction of the overall latency, ensuring that the benefits of sharding are not negated by communication overheads.
Helix Parallelism is fully compatible with modern LLM architectures, including Grouped-Query Attention (GQA) and Multi-Head Latent Attention (MLA), as well as Mixture-of-Experts (MoE) models. Its design also aligns seamlessly with emerging GPU platforms like the Blackwell system, leveraging features such as its large NVLink domains.
This new approach represents a significant step forward in making real-time inference with ultra-long-sequence LLMs practical, pushing the boundaries of throughput and latency performance. For more technical details, you can refer to the original research paper: Helix Parallelism: Rethinking Sharding Strategies for Interactive Multi-Million-Token LLM Decoding.
