TLDR: Sandwich is a novel LLM serving engine designed for CPUs that significantly improves performance by optimizing the prefill and decode phases of LLM inference separately. It achieves this through a hardware-centric approach that intelligently configures CPU core utilization (using a TopoTree abstraction) and generates highly efficient dynamic-shaped tensor programs with a fast-start-then-finetune strategy. This results in substantial throughput gains, lower latencies, and reduced tuning costs compared to existing CPU-based LLM serving solutions.
Large Language Models (LLMs) have become central to many AI applications, but serving them efficiently, especially on CPUs, presents unique challenges. While GPUs are often favored for their parallel computing power, CPUs offer a more accessible and often underutilized alternative in AI clusters. However, existing CPU-based solutions for LLM serving often fall short because they treat the entire inference process uniformly, ignoring the distinct demands of its two main phases: prefill and decode.
A new research paper, titled “Sandwich: Separating Prefill-Decode Compilation for Efficient CPU LLM Serving,” by Juntao Zhao, Jiuru Li, and Chuan Wu, introduces a novel approach to address this inefficiency. The authors propose Sandwich, a hardware-centric LLM serving engine that customizes execution plans for the prefill and decode phases, optimizing each separately.
The prefill phase involves processing the initial input sequence to generate the first token and populate the key-value (KV) cache. This phase is computationally intensive and requires handling variable-length inputs, demanding efficient, dynamic-shaped tensor programs. In contrast, the decode phase generates subsequent tokens one at a time based on the existing KV cache. While less computationally demanding, it requires extensive memory access, making it memory-bound.
Existing CPU solutions typically apply a static model partition across NUMA (Non-Uniform Memory Access) nodes and use generic vendor libraries, which is suboptimal. Sandwich recognizes that the prefill phase benefits from maximizing computational power, often utilizing all available cores, while the decode phase requires careful management of shared memory resources to avoid bottlenecks.
Sandwich tackles these challenges through two main components: service configuration generation and kernel orchestration. The service configuration generation component intelligently maps LLM serving processes to CPU cores, considering the underlying hardware topology. It uses a unique “TopoTree” abstraction to represent the CPU’s memory hierarchy, including NUMA nodes and cache structures. Through “group” and “remove” transformations on this TopoTree, Sandwich explores optimal core utilization plans, even identifying and mitigating contention on latent shared hardware structures that are not always visible to standard system tools.
The kernel orchestration component focuses on generating highly efficient tensor programs for dynamic shapes. It employs a “fast-start-then-finetune” approach for micro-kernel generation, which rapidly expands initial program shapes and then fine-tunes them for optimal performance. This method jointly optimizes computation slices and their polymerization schemes (how they are assigned to parallel computing resources). Additionally, Sandwich utilizes a “micro-kernel sliding window” and “tensor schedule reuse” to significantly reduce the time needed for tuning, especially for varying input sequence lengths.
The evaluation of Sandwich across diverse CPU platforms, including x86 (with AVX-2 and AVX-512) and ARM (with NEON), demonstrates its effectiveness. Sandwich achieved an average 2.01 times throughput improvement and significantly better time-to-first-token (TTFT) and time-per-output-token (TPOT) latencies, with up to 3.40 times lower requirements in single sequence serving. In continuous-batching serving, it showed substantial improvement in Goodput, which is the maximum request rate sustained while meeting service level objectives.
The GEMM (General Matrix Multiplication) kernels generated by Sandwich consistently outperformed representative vendor kernels and other dynamic shape solutions. Remarkably, Sandwich achieved performance comparable to static compilers while requiring three orders of magnitude less kernel tuning costs. This efficiency is crucial for adapting to the dynamic nature of LLM workloads.
Also Read:
- Optimizing LLM Inference with BucketServe’s Dynamic Batching
- Accelerating Structured Output from Large Language Models with WGrammar
By separating and optimizing the prefill and decode phases, and by deeply understanding and leveraging CPU hardware characteristics, Sandwich offers a robust and resource-friendly alternative for serving LLMs. This work highlights the significant untapped potential of CPUs for LLM inference and provides a blueprint for future hardware-aware software optimizations. You can read the full research paper here.
