TLDR: NVIDIA has introduced Dynamo, an open-source software framework designed to significantly enhance the efficiency and scalability of AI inference, particularly for large language models (LLMs). Dynamo aims to optimize GPU utilization, reduce operational costs, and accelerate the generation of tokens in AI factories, offering up to 30x faster request serving on Blackwell hardware.
NVIDIA has unveiled Dynamo, a groundbreaking open-source inference software framework poised to redefine the economics of AI inference. Launched to accelerate and scale reasoning models within AI factories, Dynamo is engineered to maximize token revenue generation for AI factories deploying reasoning AI models. It orchestrates and accelerates inference communication across potentially thousands of GPUs, succeeding the NVIDIA Triton Inference Server.
One of Dynamo’s core innovations is its disaggregated serving technique, which separates the processing (prefill) and generation (decode) phases of large language models (LLMs) onto distinct GPUs. This allows each phase to be independently optimized, catering to its specific computational needs and ensuring maximum utilization of GPU resources. This architecture is reported to double GPU throughput and improve time-to-first-token by 3x.
To achieve these significant improvements in inference performance, NVIDIA Dynamo incorporates several key features aimed at increasing throughput and reducing operational costs. The software can dynamically add, remove, and reallocate GPUs in real-time to adapt to fluctuating request volumes and types. It can also pinpoint specific GPUs within large clusters that are best suited to minimize response computations and efficiently route queries. Furthermore, Dynamo can offload inference data to more cost-effective memory and storage devices while retrieving it rapidly when required, thereby minimizing overall inference costs.
For instance, when running the DeepSeek-R1 model on a cluster of GB200 NVL72 racks, NVIDIA Dynamo demonstrated an impressive increase of over 30 times in the number of tokens generated per GPU. This level of optimization is critical as AI reasoning becomes increasingly prevalent, with each AI model expected to generate tens of thousands of tokens with every prompt, representing its ‘thinking’ process.
NVIDIA Dynamo is designed to be inference-engine agnostic, supporting various runtimes such as TRT-LLM, vLLM, and SGLang. Its modular design allows developers to select specific inference serving components, frontend API servers, and inference data transfer libraries, ensuring compatibility with existing AI stacks and avoiding costly migration efforts. The framework also includes features like the NVIDIA Dynamo Planner, Smart Router, KV Cache Block Manager, and the NVIDIA Inference Transfer Library (NIXL) to facilitate large-scale distributed and disaggregated inference serving.
Also Read:
- Langflow Empowers Local AI Agent Development on NVIDIA RTX PCs
- DataRobot and NVIDIA Unveil Agent Workforce Platform for Enterprise AI
NVIDIA’s CEO, Jensen Huang, has emphasized the importance of industries training AI models to think and learn in various ways. With NVIDIA achieving impressive revenue growth and maintaining a robust gross profit margin, Dynamo serves these models at scale, driving cost savings and enhancing efficiencies across AI factories. The intelligent inference optimizations are reported to double the performance and revenue of AI factories using the same number of GPUs on the NVIDIA Hopper platform. Dynamo is now available on GitHub, marking a significant step in revolutionizing large-scale AI model deployment efficiency.
