TLDR: HALO is a new heterogeneous accelerator that combines Compute-in-DRAM (CiD) and on-chip analog Compute-in-Memory (CiM) using 2.5D integration. It employs a phase-aware mapping strategy to optimize Large Language Model (LLM) inference, directing compute-bound prefill operations to CiM and memory-bound decode operations to CiD. This innovative approach achieves significant speedups (up to 18x over AttAcc and 2.5x over CENT) and energy reductions for low-batch, long-context LLM inference, which is critical for interactive AI applications.
Large Language Models (LLMs) are everywhere, powering everything from chatbots to personalized assistants. However, making them run efficiently, especially for interactive applications that require quick responses (known as low-batch inference), is a significant challenge. Traditional hardware often struggles with the unique demands of LLMs, which have two distinct operational phases: prefill and decode.
The prefill phase processes the initial input, which is compute-intensive, while the decode phase generates tokens one by one, making it memory-intensive. Existing accelerators tend to optimize for one phase or for high-batch scenarios, leaving a gap for low-batch, long-context applications.
Researchers Shubham Negi and Kaushik Roy from Purdue University have introduced HALO, a novel heterogeneous accelerator designed specifically to tackle these challenges. HALO is a memory-centric system that integrates two powerful technologies: Compute-in-DRAM (CiD) and on-chip analog Compute-in-Memory (CiM), all packaged together using advanced 2.5D integration. You can read the full research paper here: HALO Research Paper.
How HALO Works: A Phase-Aware Approach
HALO’s core innovation lies in its “phase-aware mapping strategy.” This means it intelligently assigns different parts of the LLM inference process to the hardware best suited for it:
- Prefill Phase (Compute-Bound): During prefill, where the model processes the entire input sequence, HALO maps these compute-heavy operations to the on-chip analog CiM. CiM excels at high-throughput matrix multiplications, making it ideal for this phase.
- Decode Phase (Memory-Bound): In the decode phase, where tokens are generated sequentially and memory access is critical, HALO utilizes CiD. By performing computations directly within the DRAM, CiD significantly reduces data movement, which is a major bottleneck in this phase.
Additionally, non-matrix operations are handled by dedicated vector and scalar units located in the HBM logic die, ensuring all aspects of LLM inference are optimized.
Why Heterogeneous Design Matters
The paper highlights the limitations of systems that rely solely on either CiD or CiM. A fully CiM system performs well in the prefill phase (6x speedup in Time-To-First-Token and 2.6x energy reduction over fully CiD), but struggles with the memory-bound decode phase. Conversely, a fully CiD system is excellent for decode (39x speedup in Time-Per-Output-Token and 3.9x energy reduction over fully CiM) but less efficient for prefill. HALO’s heterogeneous design combines the strengths of both, offering a balanced and highly efficient solution.
Also Read:
- Unpacking AI’s Energy Footprint: A Component-Level Look at Transformer Models
- Coevolutionary Continuous Discrete Diffusion: A New Paradigm for Language Models
Performance Gains
HALO was evaluated using popular LLMs like LLaMA-2 7B and Qwen3 8B. The results are impressive:
- HALO achieves up to an 18x geometric mean speedup over AttAcc, an attention-optimized mapping.
- It also delivers a 2.5x speedup over CENT, a fully CiD-based mapping.
- For the prefill phase alone, HALO1 shows a 6.54x speedup compared to CENT, especially for large input context lengths.
- Overall, HALO1 achieves a 2.4x speedup over CENT in end-to-end execution time.
- In terms of energy, HALO1 reduces energy consumption by 2x compared to AttAcc1 and 1.8x compared to CENT.
These findings demonstrate that HALO’s memory-centric and phase-aware approach significantly improves the efficiency of LLM inference, particularly for the crucial low-batch, long-context scenarios vital for interactive AI applications.
