TLDR: This research paper highlights the often-overlooked importance of CPUs in Agentic AI frameworks, which integrate LLMs with external tools. It reveals that CPU-based tool processing can account for up to 90.6% of total latency, and CPU factors significantly bottleneck throughput and energy consumption at scale. The study introduces two optimization techniques, CPU and GPU-Aware Micro-batching (CGAM) and Mixed Agentic Workload Scheduling (MAWS), which demonstrate substantial improvements in latency and efficiency for agentic AI workloads by addressing these CPU-centric challenges.
Agentic AI frameworks are transforming large language models (LLMs) from simple text generators into autonomous problem-solvers. These frameworks equip LLMs with external tools like web search, Python interpreters, and contextual databases, allowing them to plan, execute tasks, remember past steps, and adapt on the fly. While much attention has been given to the role of GPUs in AI, a recent research paper sheds light on a crucial, often overlooked aspect: the significant impact of CPUs on the performance of these agentic AI systems.
The paper, titled “A CPU-CENTRIC PERSPECTIVE ON AGENTIC AI,” by Ritik Raj, Hong Wang, and Tushar Krishna, delves into the system bottlenecks introduced by agentic AI workloads from a CPU-centric viewpoint. It systematically characterizes agentic AI based on its decision-making orchestrator, inference path dynamics, and the repetitiveness of the agentic flow, all of which directly influence system-level performance.
Understanding Agentic AI Workloads
The researchers categorized agentic AI systems along three main dimensions:
- Orchestrator-Based: This distinguishes between systems where the LLM itself controls the execution flow (LLM-orchestrated) and those where traditional programmatic code on the CPU manages tasks and tool invocation (Host-orchestrated).
- Path-Based: This differentiates between agents that follow a predetermined sequence of actions (Static Path) and those that adapt their execution based on real-time results and environmental feedback (Dynamic Path).
- Flow/Repetitiveness-Based: This looks at whether tasks are completed in a single pass (Single-step) or require iterative refinement cycles (Multi-step).
To understand these systems better, the study profiled five representative agentic AI workloads: Haystack RAG, Toolformer, ChemCrow, Langchain, and SWE-Agent. These workloads were chosen for their challenging applications, diverse computational patterns, and relevance in both academia and industry.
Demystifying CPU Bottlenecks
The profiling results revealed several key insights into where performance bottlenecks occur:
- Latency: A striking finding was that CPU-based tool processing can account for a massive portion of the total execution time—up to 90.6%. This includes tasks like data retrieval, API calls (e.g., WolframAlpha), literature searches, lexical summarization, and Python/Bash script execution. For example, in Haystack RAG, retrieval alone consumed 84.5–90.6% of the runtime. This highlights that optimizing CPUs is just as, if not more, critical than optimizing GPUs for overall latency.
- Throughput: The ability to process multiple agentic requests concurrently (throughput) was found to be bottlenecked by either CPU or GPU factors. CPU limitations included core over-subscription, cache coherence, and synchronization issues, while GPU limitations involved device memory capacity and bandwidth. The study observed that simply increasing batch size doesn’t always lead to linear throughput gains, as saturation points are reached due to these factors.
- Energy: While GPUs are often seen as the primary energy consumers in AI, the research showed that CPU dynamic energy consumption becomes significantly substantial at larger batch sizes, consuming up to 44% of the total dynamic energy. This is because CPU parallelism, especially with multi-processing, is less energy-efficient compared to GPU parallelism.
Also Read:
- Glia: An AI Architecture for Autonomous System Design and Optimization
- Unpacking AI Agents’ Skills: A New Benchmark for Tool Planning and Scheduling in Complex Tasks
Introducing Key Optimizations
Based on these insights, the researchers proposed two main scheduling optimizations:
- CPU and GPU-Aware Micro-batching (CGAM): This technique addresses throughput saturation by capping the batch size and processing micro-batches sequentially. CGAM can lead to significant improvements in P50 latency (up to 2.1x speedup), reduce KV cache usage on GPUs by almost half, and substantially cut down CPU dynamic energy consumption. An advanced version, CGAMoverlap, further optimizes by overlapping CPU and GPU tasks for even better P90 latency.
- Mixed Agentic Workload Scheduling (MAWS): Recognizing that agentic workloads can be heterogeneous (some CPU-heavy, some LLM-heavy), MAWS adaptively uses multi-processing for CPU-heavy tasks and multi-threading for LLM-heavy tasks. This approach prevents CPU over-subscription for LLM-heavy tasks, freeing up resources and making CPU-heavy tasks more effective.
The evaluation demonstrated that CGAM and MAWS, both individually and combined, offer substantial performance and efficiency gains. For instance, CGAM achieved up to 2.1x P50 latency speedup for homogeneous workloads, and MAWS+CGAM provided a 2.1x P50 latency speedup for CPU-heavy tasks in mixed workloads, along with overall P99 latency savings.
This research underscores the critical need for a holistic, CPU-centric approach to optimizing agentic AI systems, moving beyond a sole focus on GPUs. By understanding and addressing CPU bottlenecks, developers can unlock significant improvements in the performance, efficiency, and scalability of these advanced AI frameworks. You can read the full paper here.
