TLDR: NVIDIA researchers have demonstrated that extended reinforcement learning, combined with verifiable rewards, enhanced Group Relative Policy Optimization (GRPO), and stability techniques like KL regularization and periodic policy resets, significantly boosts the reasoning capabilities of a small language model (Nemotron-Research-Reasoning-Qwen-1.5B). This approach led to substantial performance gains across diverse tasks including math, coding, logic puzzles, STEM reasoning, and instruction following, achieving results competitive with larger or specialized models. The model and methodology are open-sourced to encourage further research.
Recent advancements in large language models (LLMs) have shown remarkable progress in complex reasoning tasks like mathematics and code generation. This progress is often attributed to scaling up test-time computation, such as using chain-of-thought reasoning, and leveraging large-scale reinforcement learning (RL), especially when combined with reward signals that can be objectively verified.
NVIDIA researchers have explored the effects of prolonged reinforcement learning on a smaller language model, aiming to unlock diverse reasoning capabilities. Their work identifies several crucial elements for effective training, including the use of verifiable reward tasks, improvements to an optimization algorithm called Group Relative Policy Optimization (GRPO), and practical techniques to enhance training stability and generalization.
Key Ingredients for Enhanced Training
The study highlights that using tasks with “verifiable rewards” is essential. Unlike learned reward models, which can be susceptible to generating overly polite or verbose but incorrect answers (known as reward hacking), verifiable rewards provide direct, grounded feedback on correctness. This ensures that the model learns to solve problems accurately rather than just appearing to do so.
The researchers also introduced enhancements to GRPO, including “decoupled clipping” and “dynamic sampling.” Decoupled clipping helps maintain the diversity of the model’s outputs, preventing it from prematurely narrowing its focus. Dynamic sampling improves efficiency by concentrating training on examples that offer the most learning signal, filtering out those the model already consistently solves or fails.
To ensure training stability and continued improvement over long periods, they implemented “controlled KL regularization” and “periodic reference policy resets.” KL regularization helps prevent the model’s output distribution from becoming too narrow (a phenomenon called entropy collapse), while periodic resets of the reference policy allow the model to continue learning and adapting without being overly constrained by its initial state.
Diverse Training Data and Impressive Gains
A significant aspect of this research is the use of a wide variety of training data, all with verifiable reward signals. This includes traditional reasoning domains like mathematics and code generation, as well as more complex areas such as STEM problem-solving, logical puzzles, and instruction following. This diversity helps the model generalize its reasoning abilities across different types of problems.
The results are compelling: the model achieved substantial improvements over strong baselines. For instance, it showed a +14.7% improvement on math tasks, +13.9% on coding, and a remarkable +54.8% on logic puzzle tasks. It also saw gains of +25.1% on STEM reasoning and +18.1% on instruction-following. The model, named Nemotron-Research-Reasoning-Qwen-1.5B, is publicly released to support further research.
Training Methodology
The training involved a staged strategy, iteratively refining the model’s behavior, incorporating new data, and adjusting settings. They used a base model called DeepSeek-R1-Distill-Qwen-1.5B and conducted training on powerful NVIDIA-H100-80GB nodes, accumulating approximately 16,000 GPU hours.
The researchers also performed “ablation studies” to understand the contribution of each component. They found that a higher “rollout sampling temperature” was beneficial for exploration throughout training, preventing the model from getting stuck in suboptimal patterns. Decoupled clipping and dynamic sampling consistently improved performance and stability. The periodic resetting of the reference policy proved crucial for overcoming performance plateaus and maintaining training stability over extended periods.
Also Read:
- Boosting LLM Reasoning: A New Approach with Question Augmentation
- Unlocking Advanced Reasoning in Large Language Models Through Small Model Guidance
Conclusion and Future Impact
This research demonstrates that even smaller language models can achieve significant reasoning improvements through carefully designed and prolonged reinforcement learning. The techniques developed, such as decoupled clipping, dynamic sampling, controlled KL regularization, and periodic reference policy resets, are critical for stable and effective training across diverse tasks. The approach allowed their model to achieve competitive performance with larger or domain-specialized models, despite being trained on a broader range of tasks.
By open-sourcing their model and sharing their training methodology, the NVIDIA team aims to foster further advancements in the field of resource-efficient language models. You can find the full research paper here: Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training.
