TLDR: xLLM is an intelligent and efficient inference framework for Large Language Models (LLMs) designed for large-scale enterprise serving. It tackles challenges like dynamic workloads, underutilized AI accelerators, and distributed storage by introducing a decoupled service-engine architecture. This includes intelligent scheduling, dynamic resource allocation, global KV Cache management, multi-layer execution pipelines, adaptive graph modes, and algorithmic optimizations, leading to significantly improved throughput and resource efficiency compared to existing frameworks.
In the rapidly evolving landscape of artificial intelligence, Large Language Models (LLMs) are becoming central to many enterprise applications, from chatbots to recommendation systems. However, efficiently running these massive models at scale presents significant challenges. Existing inference frameworks often struggle with dynamic workloads, underutilized hardware, and complex distributed environments.
A new framework called xLLM has emerged to address these critical issues. Developed by a team of researchers from JD.com, THU, USTC, BUAA, and PKU, xLLM is designed as an intelligent and efficient LLM inference framework specifically for high-performance, large-scale enterprise-grade serving, with deep optimizations for various AI accelerators.
The Core Problem: Why Current Systems Fall Short
The current generation of LLM inference frameworks faces several practical hurdles. Enterprise-grade serving environments deal with a mix of online and offline tasks, demanding high availability and efficient distributed storage. Simultaneously, the sheer scale and complexity of modern LLMs, coupled with new hardware paradigms and inference algorithms, often lead to AI accelerators being underutilized, creating performance bottlenecks.
xLLM’s Innovative Architecture: Decoupled Service and Engine
xLLM tackles these challenges with a novel decoupled service-engine architecture, splitting its functionalities into two main components: xLLM-Service and xLLM-Engine.
xLLM-Service: Intelligent Orchestration at Scale
The service layer, xLLM-Service, acts as the intelligent brain, managing instances and scheduling requests efficiently. Key innovations include:
- Unified Elastic Scheduling: It intelligently handles both online (latency-sensitive) and offline (batch-oriented) tasks, co-locating them to maximize cluster utilization without compromising the strict performance requirements of online services.
- Dynamic Prefill-Decode (PD) Disaggregation: Traditional systems often use static resource allocation for the prefill and decode phases of LLM inference, which is inefficient for fluctuating workloads. xLLM-Service dynamically adjusts resource allocation based on real-time demand, ensuring optimal utilization and responsiveness.
- Hybrid Encode-Prefill-Decode (EPD) for Multimodal Inputs: For multimodal LLMs that process images, voice, and text, xLLM introduces a novel EPD disaggregation policy. This allows for intelligent resource allocation across the encoding, prefill, and decode phases, optimizing performance for complex inputs.
- Global KV Cache Management: As LLMs grow, the Key-Value (KV) Cache consumes vast amounts of memory. xLLM implements a distributed architecture for global KV Cache management, improving memory handling and cache hit rates across multiple accelerators.
- Fast Fault Recovery: To ensure high availability, xLLM-Service includes a robust fault-tolerant system that can quickly detect and recover from failures, minimizing service interruptions.
xLLM-Engine: Maximizing Hardware Performance
The engine layer, xLLM-Engine, focuses on co-optimizing system and algorithm designs to fully saturate computing resources. Its advancements include:
- Multi-layer Execution Pipeline: xLLM-Engine optimizes execution at multiple levels. It overlaps CPU scheduling with AI accelerator operations to reduce idle time, uses dual-stream parallelism to overlap computation with communication, and fine-tunes computational units for maximum hardware utilization.
- Adaptive Graph Mode: To reduce overhead from frequent kernel launches, xLLM-Engine uses an adaptive graph mode. This technique pre-compiles sequences of operations into a single computational graph, significantly speeding up execution, especially for dynamic input shapes.
- Efficient Memory Management (xTensor): xLLM introduces a unique “logically contiguous, physically discrete” xTensor memory management scheme for the KV Cache. This approach resolves conflicts between memory contiguity requirements and dynamic allocation needs, leading to both high computational efficiency and memory utilization.
- Algorithmic Enhancements: The framework integrates advanced algorithms like optimized speculative decoding for faster token generation, dynamic Expert Parallel Load Balance (EPLB) for Mixture-of-Experts (MoE) models, and hierarchical Data Parallel (DP) load balance to ensure even workload distribution across computing units.
- Generative Recommendation Optimizations: For specific applications like generative recommendation, xLLM provides scenario-specific optimizations, including host-side beam search improvements and device-side valid item filtering, enhancing both accuracy and efficiency.
Also Read:
- KVCOMM: Boosting Multi-Agent LLM Efficiency with Smart Memory Reuse
- Elastic-Cache: Smarter Decoding for Diffusion Language Models
Real-World Impact and Performance
Extensive evaluations demonstrate xLLM’s superior performance. It achieves throughput up to 1.7 times that of MindIE and 2.2 times that of vLLM-Ascend with Qwen-series models, and an average of 1.7 times that of MindIE with Deepseek-series models, all while maintaining strict performance constraints. xLLM is already deployed in production at JD.com, supporting core business scenarios like the JingYan AI chatbot, marketing recommendations, product understanding, and customer service assistants.
The framework’s open-source release aims to foster further innovation in enterprise-scale inference solutions, hardware optimization, and integrated service-engine architectures for next-generation AI applications. For more in-depth details, you can read the full technical report here.
