TLDR: CoPRIS is a new reinforcement learning framework designed to make training large language models (LLMs) faster and more stable. It addresses the inefficiencies of traditional synchronous RL systems, which often waste computational resources waiting for long responses. CoPRIS achieves this by managing a fixed number of concurrent rollouts, early-terminating when enough samples are collected, and reusing unfinished work. It also introduces a ‘Cross-stage Importance Sampling Correction’ to ensure stable training even when using data generated by older policy versions. Experiments show CoPRIS can train LLMs up to 1.94 times faster while maintaining or improving performance on mathematical reasoning tasks, and it scales well with context length and model size.
Reinforcement Learning (RL) has become a pivotal method for enhancing the capabilities of large language models (LLMs), especially in complex tasks like mathematical reasoning and code generation. However, a significant challenge in current RL systems for LLMs is their inefficiency, primarily due to a fully synchronous design. This means that the training process often grinds to a halt, waiting for an entire batch of responses (known as ‘rollouts’) to complete, even if some responses are exceptionally long. This ‘long-tail problem’ leads to many powerful GPUs sitting idle, wasting valuable computational resources.
Introducing CoPRIS: A Leap in RL Efficiency
To tackle this widespread inefficiency, researchers have introduced a novel framework called Concurrency-Controlled Partial Rollout with Importance Sampling, or CoPRIS. This innovative approach aims to significantly speed up RL training for LLMs without compromising their performance or stability. CoPRIS addresses the core issues of synchronous systems by rethinking how rollouts are managed and how off-policy data is handled.
How CoPRIS Works Its Magic
CoPRIS builds upon the concept of ‘partial rollout’ but introduces two crucial mechanisms to ensure both efficiency and stability:
First, **Concurrency-Controlled Generation** is at its heart. Instead of waiting for all rollouts in a batch to finish, CoPRIS maintains a fixed number of concurrent rollouts. As soon as one trajectory is completed, a new one is immediately started, ensuring that computational resources are always busy. This controlled concurrency also prevents memory overload, which can cause costly recomputations, and helps manage the amount of ‘off-policy’ data – data generated by an older version of the model’s policy.
Second, CoPRIS employs **Cross-stage Importance Sampling Correction**. In asynchronous systems, some parts of a trajectory might be generated by an older policy while others are generated by the current one. Directly training on this mixed data can lead to instability. CoPRIS solves this by storing the ‘log probabilities’ (a measure of how likely the model was to generate a particular token) for each token when it was created. During training, these stored log probabilities are combined with those from the current policy to apply an ‘importance sampling correction.’ This technique effectively reweights the data, allowing the system to learn accurately even from off-policy trajectories, thus maintaining training stability.
Additionally, CoPRIS incorporates **Early Termination** and **Buffering of Partial Trajectories**. Once enough samples are collected for a training step, the generation process halts. Any unfinished trajectories are not discarded but are stored in a buffer along with their log probabilities. These partial trajectories are then prioritized for resumption in subsequent rollouts, further reducing wasted computation.
Impressive Results and Scalability
Experiments conducted on challenging mathematical reasoning benchmarks, using various LLM sizes, have shown remarkable results. CoPRIS achieved an impressive speedup of 1.58 to 1.94 times faster training compared to traditional synchronous RL systems. Crucially, it managed to do this while maintaining comparable or even superior performance. For instance, on some models, CoPRIS showed average score increases, demonstrating that efficiency gains do not come at the cost of model quality.
The framework also demonstrates excellent scalability. As context lengths (the amount of text an LLM can process at once) increased from 8,000 to 40,000 tokens, CoPRIS’s speedup progressively grew, highlighting its effectiveness in long chain-of-thought reasoning scenarios. Similarly, it consistently outperformed baselines across different model sizes (1.5B, 7B, and 14B parameters), proving its robustness.
Also Read:
- SofT-GRPO: Advancing LLM Reasoning with Gumbel-Reparameterized Soft-Thinking
- AI Models Learn from Mistakes with New Self-Correction Method
A Step Towards More Efficient AI Training
CoPRIS represents a significant advancement in the field of reinforcement learning for large language models. By intelligently managing concurrent rollouts and applying sophisticated importance sampling, it effectively mitigates the long-standing inefficiencies of synchronous training. This work paves the way for building more efficient and stable RL training frameworks, ultimately accelerating the development of more capable and intelligent AI systems. You can explore the full research paper for more details at arXiv:2511.05589.
