TLDR: VoltanaLLM is a new system that significantly reduces the energy consumption of Large Language Model (LLM) inference by up to 36.3% while maintaining high performance. It achieves this by intelligently adjusting GPU frequencies for different processing phases (prefill and decode) and routing requests to avoid inefficiencies, all guided by a lightweight latency predictor. This makes LLM serving more sustainable and cost-effective.
Large Language Models (LLMs) are at the heart of many interactive AI applications today, from chat assistants to code generation tools. However, the immense energy consumption required for running these models poses a significant challenge for both cost-effectiveness and environmental sustainability. A new research paper introduces VoltanaLLM, an innovative system designed to make LLM serving more energy-efficient while ensuring a smooth user experience.
The core problem VoltanaLLM addresses is how to reduce the energy footprint of LLM inference without compromising on performance metrics like Time-To-First-Token (TTFT) and Inter-Token Latency (ITL), which are crucial for real-time applications. The researchers observed that simply lowering GPU frequency doesn’t always save energy; instead, there’s a “U-shaped” energy-frequency curve, indicating an optimal frequency point for energy efficiency. This optimal point also varies between the two main phases of LLM inference: prefill (processing the input prompt) and decode (generating output tokens).
Another key insight was the dynamic nature of real-world LLM workloads. The demand for prefill and decode operations fluctuates significantly throughout the day, meaning a one-size-fits-all frequency setting is inefficient. Furthermore, the team discovered that specific batch sizes (the number of requests processed together) can lead to GPU underutilization and energy waste, particularly during the decode phase.
VoltanaLLM tackles these challenges by adopting a control theory perspective and leveraging a disaggregated architecture where prefill and decode operations are handled by separate GPU instances. The system comprises three main components:
EcoFreq: Smart Frequency Control
EcoFreq is a feedback-driven controller that independently adjusts the GPU frequency for each prefill and decode instance. It operates on a per-iteration basis, meaning it can react quickly to changing workloads. By running in a separate process, it minimizes overhead. EcoFreq’s goal is to find the lowest possible frequency that still meets the specified latency targets (SLOs), thus maximizing energy savings.
EcoRoute: Intelligent Request Routing
For decode instances, VoltanaLLM introduces EcoRoute, a state-space navigation-based router. This component intelligently dispatches requests to different decode instances. Instead of simply balancing the load, EcoRoute performs a “what-if” analysis to predict how adding a request would affect an instance’s frequency and energy efficiency. It aims to avoid pushing instances across inefficient batch size boundaries, allowing some instances to operate at lower, more energy-efficient frequencies.
Also Read:
- Enhancing Language Model Reasoning with Dynamic Confidence Assessment
- AI Agents Team Up to Accelerate GPU Code
EcoPred: Accurate Latency Prediction
Both EcoFreq and EcoRoute rely on EcoPred, a lightweight and accurate latency predictor. EcoPred uses simple linear regression models, trained on profiling data, to estimate TTFT and ITL based on factors like batch size and tokens in KV cache. This allows VoltanaLLM to make fast, informed decisions about frequency scaling and request routing without complex, resource-intensive models.
The researchers implemented VoltanaLLM on SGLang, a popular LLM inference engine, and tested it with various state-of-the-art LLMs and real-world datasets. The results are impressive: VoltanaLLM achieved up to 36.3% energy savings compared to systems running at maximum frequency, all while maintaining near-perfect SLO attainment rates. This demonstrates a significant step towards more sustainable and cost-effective deployment of large language models.
For more in-depth technical details, you can read the full research paper here.
