TLDR: HyperFlexis is a new LLM serving system that integrates advanced scheduling and rapid scaling to efficiently handle varied requests with different performance targets (SLOs). It features a multi-SLO-aware scheduler, supports both collocated and disaggregated prefill/decode architectures, and uses a device-to-device weight transfer for fast scaling. This leads to significantly higher SLO attainment, lower latency, and cost efficiency compared to existing systems.
Large Language Models (LLMs) are everywhere, powering everything from chatbots to complex document summarization. But making these powerful models work smoothly for everyone, especially when different applications have different needs, is a big challenge. Imagine a chatbot needing an instant reply versus a summarization tool that can take a bit longer but needs to process a lot of text quickly. This is where Service Level Objectives (SLOs) come in – they are the performance targets a system aims to meet, like how fast the first word appears (Time-to-First-Token or TTFT) or how quickly subsequent words are generated (Time-Per-Output-Token or TPOT).
The problem is, current LLM serving systems struggle to handle these diverse demands efficiently. They face issues with highly varied requests, different priorities, and the need to scale up or down quickly and affordably. This is particularly complex when dealing with both “collocated” systems (where all processing happens on the same device) and “disaggregated” systems (where different parts of the process, like prefill and decode, are handled separately for better scaling).
Introducing HyperFlexis
A new system called HyperFlexis has been developed to tackle these challenges head-on. It’s a unified LLM serving system that brings together smart algorithms and system-level innovations to optimize how requests are scheduled and how resources are scaled, all while keeping multiple SLOs in mind. Think of it as a conductor for a complex orchestra, ensuring every instrument plays its part perfectly to meet the audience’s expectations.
One of HyperFlexis’s core features is its multi-SLO-aware scheduler. This scheduler is like a smart traffic controller, using budget estimations and prioritizing requests to make sure that both new and ongoing tasks meet their performance targets proactively. For systems where prefill (generating the first part of the response) and decode (generating the rest) are handled separately, HyperFlexis offers specialized scheduling for each stage, even managing the transfer of data between them efficiently.
Smart Scaling and Speed
Beyond scheduling, HyperFlexis also excels at cost-effective scaling. It can quickly decide when to add or remove resources, and it intelligently links prefill and decode instances during these scaling operations. A standout innovation is its rapid P/D role transitions, allowing devices to switch between prefill and decode tasks seamlessly.
To make scaling even faster and reduce the dreaded “cold-start latency” (the delay when a new system starts up), HyperFlexis introduces a device-to-device (D2D) weight transfer mechanism. Instead of loading large model weights from storage every time, it can transfer them directly from an already running instance. This dramatically cuts down the weight loading overhead by up to 19.39 times, meaning new instances can be ready almost instantly.
How It Works: A Closer Look
HyperFlexis uses a centralized dispatcher that manages incoming requests, categorizing them by their SLOs and placing them in priority queues. It then selects the most suitable worker based on its availability and calculates a “token budget” – the maximum number of tokens a worker can generate without violating any SLOs. This ensures that urgent requests are handled promptly without overloading the system.
For scenarios where users can’t specify exact latency targets, HyperFlexis includes a clever priority-based SLO mapping scheme. Users can simply assign a priority (e.g., high, medium, low) to their applications, and the system dynamically translates these into specific latency SLOs, adapting them over time based on real-world performance.
The scaling mechanism is threshold-based, meaning it monitors system metrics like worker utilization and request wait times. If these metrics go above a certain point, more workers are added; if they stay below a certain point for too long, workers are removed to save costs. In disaggregated setups, prefill and decode phases are scaled independently, and workers can even dynamically switch roles between them.
Also Read:
- CommonKV: A Training-Free Approach to Efficient LLM Memory Management
- Optimizing Federated Learning in Competitive Multi-Service Environments
Impressive Results
The evaluation of HyperFlexis shows significant improvements. It achieves up to 4.44 times higher SLO attainment, meaning more requests meet their performance targets. It also reduces request latency by up to 65.82% and maintains cost parity with, or even lowers costs compared to, state-of-the-art systems. These benefits are observed across various LLM models and complex multi-task workloads, in both collocated and disaggregated architectures.
HyperFlexis represents a significant step forward in making LLM serving more efficient, responsive, and cost-effective, especially for diverse and demanding real-world applications. The research paper, titled “HyperFlexis: Joint Design of Algorithms and Systems for Multi-SLO Serving and Fast Scaling,” provides a comprehensive look at this innovative system. You can read the full paper here.
