TLDR: Equinox is a new scheduling system for Large Language Models (LLMs) that addresses the limitations of existing methods in fairly allocating resources. It uses a dual-counter framework, combining a User Fairness Counter (UFC) for user experience (latency, weighted tokens) and a Resource Fairness Counter (RFC) for operational efficiency (throughput, GPU utilization). To enable proactive scheduling, Equinox employs a Mixture of Prediction Experts (MoPE) to accurately estimate these metrics before execution. Evaluations show Equinox significantly improves throughput, reduces latency, and enhances fairness compared to previous systems like VTC, while maintaining high GPU utilization.
Large Language Models (LLMs) have rapidly transformed artificial intelligence, becoming essential for various tasks from content creation to software engineering. This widespread adoption has led to an unprecedented demand on the systems that serve these models. However, a new research paper titled “Equinox: Holistic Fair Scheduling in Serving Large Language Models” highlights a significant challenge: existing scheduling methods struggle to allocate resources fairly across the diverse and complex workloads LLMs generate.
Traditional scheduling approaches, like First-Come-First-Served (FCFS), offer no protection against a single client monopolizing resources, leading to poor service for others. Requests Per Minute (RPM) policies, while attempting to limit usage, often waste valuable resources during periods of low demand. Even more advanced systems like the Virtual Token Counter (VTC) fall short because they rely on a single metric – token consumption – which doesn’t accurately capture the full resource dynamics of LLMs.
The core issue lies in the unique architecture of Transformer models, which power LLMs. These models have two distinct phases: ‘prefill’ and ‘decode’. Prefill operations, which process the initial prompt, are compute-intensive and determine overall throughput. Decode operations, which generate the response token by token, are memory-intensive and largely dictate latency. These conflicting resource patterns mean that a single metric, like token count, cannot adequately define fairness. For instance, two requests with the same total token count might have vastly different impacts on system resources and user experience.
Introducing Equinox: A Dual-Counter Approach to Fairness
To address these limitations, researchers from Shanghai Jiao Tong University and other institutions have developed Equinox, a novel open-source system that introduces a dual-counter framework for holistic fair scheduling. This framework considers fairness from two crucial perspectives: the user and the operator.
The User Fairness Counter (UFC) measures the quality of service experienced by the user. It combines weighted tokens (giving more importance to output tokens, which are more resource-intensive) with user-perceived latency, including both wait time and predicted processing time. This ensures that users receive equitable service quality.
The Resource Fairness Counter (RFC) focuses on operational efficiency. It tracks system throughput (tokens processed per second) and GPU utilization. This helps ensure that the server’s resources are used effectively and equitably from the operator’s viewpoint.
Equinox then combines these two counters into a single “Holistic Fairness” score, using tunable parameters to balance user experience and system efficiency. By prioritizing requests from users with the lowest holistic fairness score, Equinox aims to achieve a balanced quality of service for all.
Overcoming the Prediction Paradox with MoPE
A significant challenge in implementing this dual-counter framework is that the metrics needed for fairness calculation (latency, output tokens, throughput, GPU utilization) are typically only known after a request has been processed. This creates a “scheduling paradox.” Equinox solves this with its deterministic Mixture of Prediction Experts (MoPE) framework.
MoPE is a lightweight prediction system that routes incoming requests to specialized “experts” based on their characteristics. For example, one expert might handle short requests, another medium, and another long ones. These experts provide low-overhead estimates for all four fairness components (user-perceived latency, output tokens, GPU utilization, and throughput) before a request even enters the GPU pipeline. This proactive prediction capability is crucial for enabling Equinox’s fairness-aware scheduling.
Also Read:
- HyperFlexis: Optimizing LLM Serving for Diverse Performance Needs
- Scaling Large Language Model Inference on HPC Clusters with SLURM
Performance and Impact
The Equinox system also incorporates other optimizations like adaptive batching and stall-free scheduling. Evaluations conducted on real-world production traces (ShareGPT, LMSYS Chat-1M) and synthetic workloads have demonstrated impressive results. Equinox achieved up to 1.3 times higher throughput, 60% lower time-to-first-token latency, and 13% higher fairness compared to VTC, all while maintaining a high 94% GPU utilization. These findings confirm that Equinox can deliver fairness under varying conditions across different platforms.
In conclusion, Equinox offers a sophisticated and effective solution to the complex problem of fair scheduling in LLM serving. By moving beyond single-metric approaches and embracing a holistic view of fairness, it ensures a better experience for users and more efficient resource utilization for operators. For more technical details, you can refer to the full research paper: Equinox: Holistic Fair Scheduling in Serving Large Language Models.
