TLDR: RLinf is a new system for training large-scale reinforcement learning (RL) models more efficiently. It addresses the challenges of diverse and dynamic RL workflows by introducing a “macro-to-micro flow transformation” (M2Flow) paradigm. This allows developers to define high-level RL tasks while the system automatically optimizes how these tasks are executed on hardware, using techniques like flexible pipelining and context switching. RLinf achieves significant speedups (1.1x to 2.13x) over existing systems for both reasoning and embodied RL tasks, and its open-source nature aims to accelerate RL innovations.
Reinforcement learning (RL) is a powerful technology driving advancements in artificial general intelligence, agentic intelligence, and embodied intelligence. However, the diverse and ever-changing nature of RL tasks often leads to inefficient use of hardware and slow training times on current systems. This inefficiency stems from a lack of flexibility in how these systems handle complex RL workflows.
A new system called RLinf has been introduced to tackle these challenges. RLinf is designed to provide high-performance RL training by focusing on system flexibility. Its core innovation is a novel design approach called ‘macro-to-micro flow transformation’ (M2Flow). This paradigm allows developers to define high-level, easy-to-understand RL workflows, which RLinf then automatically breaks down and reassembles into highly optimized execution flows across both time and space.
The problem with existing RL training systems is their inability to adapt to the varied characteristics of different RL components. For instance, some parts of an RL workflow, like training a large language model (LLM), require significant GPU memory for gradients and optimizer states. Other parts, like LLM generation, might underutilize GPUs due to memory bandwidth bottlenecks. Furthermore, components like embodied simulators often rely heavily on CPUs for physics calculations and GPUs for 3D rendering, demanding different resource allocation strategies. Simple execution modes, such as running all components sequentially (collocated execution) or fully separating them onto different accelerators (disaggregated pipelining), often lead to inefficiencies like idle hardware or memory imbalances.
RLinf addresses these issues by decoupling the logical programming of RL workflows from their physical execution planning. This means developers can write clear, intuitive workflows without needing to worry about the intricate details of how they will be executed on the hardware. RLinf then takes this logical flow and transforms it into a fine-grained execution plan tailored to the specific workload and available hardware.
How RLinf Achieves Flexibility and Efficiency
RLinf employs three key mechanisms to realize its M2Flow transformation:
-
Worker Abstraction and Adaptive Communication: RLinf encapsulates each RL component as a ‘worker.’ These workers can be flexibly placed on different hardware and communicate directly and efficiently with each other, regardless of where they are located or how data is arranged.
-
Elastic Pipelining and Automatic Context Switching: These mechanisms expand the system’s scheduling capabilities. Elastic pipelining allows workers to process data at varying granularities, enabling flexible pipelining. Automatic context switching allows different workers to share the same hardware sequentially, especially when they cannot co-reside due to memory limitations. This is managed through a distributed device lock that ensures exclusive access to resources and automatically loads/offloads worker resources as needed.
-
Profiling-Guided Scheduling Policy: RLinf includes a profiler that measures the execution time and memory usage of each component under different conditions. This information is then fed to a scheduler, which uses it to automatically determine the most efficient execution mode, including GPU assignments, pipelining configurations, and data processing granularity.
The system supports a wide range of execution modes, from pure temporal scheduling (workers taking turns on all accelerators) to pure spatial scheduling (workers on separate GPUs with pipelining), and even hybrid modes that combine both. This adaptability allows RLinf to optimize performance for diverse RL workloads.
Also Read:
- DiffusionNFT: A Faster, More Flexible Way to Train Generative AI Models
- Enhancing Reinforcement Learning with Adaptive Demonstration Guidance
Performance and Impact
Extensive evaluations show that RLinf consistently outperforms state-of-the-art RL training systems. It achieves a speedup of 1.1 times to 2.13 times in end-to-end training throughput. For reasoning RL tasks using models like Qwen2.5, RLinf demonstrated significant throughput improvements compared to baselines like veRL. In embodied RL training, RLinf showed substantial speedups in both hybrid and collocated modes, depending on whether the task was GPU-bound or CPU-bound.
Beyond raw speed, models trained with RLinf also achieved superior algorithmic performance on various benchmarks. For example, new models like RLinf-math-1.5B and RLinf-math-7B outperformed comparable open-source models on math benchmarks. Similarly, embodied RL models trained with RLinf achieved state-of-the-art success rates on complex tasks in environments like ManiSkill and LIBERO.
RLinf is implemented in Python and leverages Ray for cluster management. It supports a variety of RL algorithms and models, including popular ones like PPO and GRPO, and models like Qwen and OpenVLA. The codebase is open-sourced, aiming to accelerate RL innovations in the era of large language models.
This work represents a significant step towards more flexible and efficient AI runtimes, offering a blueprint for systems that can intelligently orchestrate diverse components like training, inference, simulation, and reasoning within a unified framework. For more technical details, you can refer to the full research paper here.
