TLDR: Continuum is a new LLM serving system that significantly improves the efficiency of multi-turn agentic applications. It addresses challenges like KV cache eviction and scheduling delays caused by tool calls by introducing a ‘tool-aware KV cache time-to-live’ mechanism. By predicting tool call durations and intelligently pinning KV caches, Continuum reduces repetitive computations and ‘scheduling bubbles,’ leading to faster job completion times and higher throughput for complex agent workflows.
Large Language Models (LLMs) are rapidly evolving beyond simple text generation, now powering sophisticated ‘agentic’ applications. These agents can reason, plan, and interact with external tools, making them incredibly versatile for tasks like software engineering, information retrieval, and scientific research. However, serving these complex agentic workflows efficiently has presented significant challenges for existing LLM serving systems.
Traditional LLM serving systems, like vLLM and SGLang, were primarily designed for single-turn text generation. They struggle with agentic applications because these workflows involve multiple turns, where an LLM generates a thought, calls an external tool (like a code interpreter or a web search), and then processes the tool’s output before generating the next thought. These tool calls introduce pauses, breaking the continuous flow of LLM requests.
The core problems identified are two-fold. First, during these pauses, existing systems often evict the LLM’s Key-Value (KV) cache from GPU memory to free up resources. When the agent needs the LLM again after a tool call, this eviction leads to costly ‘repetitive prefill or loading’ of the KV cache. Second, even if the KV cache is offloaded to CPU memory, the subsequent LLM request might have to wait for GPU memory to become available, creating ‘scheduling bubbles’ that significantly increase the overall time it takes to complete an agent’s task, especially in multi-turn scenarios.
A new research paper introduces a novel serving system called Continuum, designed to tackle these inefficiencies head-on. Continuum optimizes job completion time for multi-turn agent workloads by intelligently managing the KV cache and implementing program-level scheduling. The central idea is to use a ‘tool-aware KV cache time-to-live (TTL)’.
How Continuum Works
Continuum predicts the duration of tool calls within an agent’s workflow. Based on this prediction and the total number of turns in an agent’s program, it selectively ‘pins’ the KV cache in GPU memory for a specific TTL value. This means that if a tool call is expected to be short, Continuum keeps the KV cache readily available. If the tool call returns before the TTL expires, the next LLM request can immediately resume without needing to re-prefill or reload the KV cache, thus eliminating repetitive computations and scheduling bubbles.
The system maintains statistics on tool call durations, both globally and for individual tools, updating them online as new data becomes available. This allows Continuum to make increasingly accurate predictions for setting the TTL. It also incorporates the number of turns in a program into its TTL calculation, recognizing that multi-turn programs benefit more from sustained KV cache availability.
Continuum also introduces a ‘pinning-aware priority’ in its scheduler. This prioritizes requests whose KV caches are already pinned, ensuring that these programs continue their execution as smoothly as possible. This approach preserves the natural order of an agent’s multi-turn program, connecting LLM requests as much as possible and reducing overall latency.
Also Read:
- Agentic AI’s Hidden Engine: The CPU’s Critical Role in Performance
- GLM: Enhancing LLM Reasoning on Graphs with Multi-Agent Collaboration and Optimized Serving
Performance and Impact
Evaluations on real-world agentic workloads, such as SWE-Bench (software engineering tasks) and BFCL (function calling tasks), using Llama-3.1 8B and 70B models, demonstrate Continuum’s effectiveness. The system significantly improves average job completion times, reducing delays by 1.12x to 3.66x and boosting throughput by 1.10x to 3.22x compared to state-of-the-art baselines. These improvements hold true across various hardware setups and even with DRAM offloading schemes, highlighting Continuum’s fundamental advancements in scheduling and KV cache management.
The robustness of Continuum is particularly evident in scenarios with an increasing number of turns, where traditional methods degrade significantly. Continuum maintains stable, low-latency performance, proving its suitability for complex, long-horizon agentic interactions. The system also introduces only minor scheduling overhead, which is negligible compared to the substantial performance gains.
This innovation has broader implications beyond just online serving. It can accelerate post-training procedures like reinforcement learning from verifiable rewards, where the rollout stage (model interaction with environments) is a dominant cost. By increasing inference throughput, Continuum can reduce the number of GPUs needed for rollout generation, leading to significant savings in training time and compute costs.
For more technical details, you can refer to the full research paper: CONTINUUM: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live.
