TLDR: This paper resolves the debate on whether Reinforcement Learning with Verifiable Rewards (RLVR) shrinks or expands LLM reasoning capabilities by proposing a two-stage dynamic. Initially, RLVR leads to “exploitation” and potential capability shrinkage. With prolonged training, it transitions to an “exploration” stage, enabling genuine capability expansion. The authors introduce a method using “relative negative gradients” to facilitate this prolonged training and expansion, demonstrating improved performance and diversity in LLMs.
The world of artificial intelligence, particularly large language models (LLMs), is constantly evolving. A key technique used to enhance these models’ abilities is Reinforcement Learning with Verifiable Rewards (RLVR). This method helps LLMs excel at complex tasks like mathematics and programming by optimizing them with clear reward signals. However, there’s been a significant debate: does RLVR truly make LLMs smarter, expanding their reasoning capabilities, or does it actually narrow them down?
A recent research paper, “The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic View,” dives deep into this question. The authors, Xinhao Yao, Lu Yu, Xiaolin Hu, Fengwei Teng, Qing Cui, Jun Zhou, and Yong Liu, propose a fascinating perspective that suggests both sides of the debate are partially correct. They introduce a “two-stage dynamic” to explain how RLVR impacts LLM reasoning.
The Two Stages of RLVR Impact
The first stage is called the “Exploitation Stage.” In this initial phase of training, the language model tends to focus on what it already knows works well (high-reward tokens) and avoids what doesn’t (low-reward tokens). It rarely explores new, potentially optimal solutions. This intense focus on known good options can lead to a “shrinkage” of the model’s capability boundary. Essentially, it becomes very good at what it’s already familiar with, but less diverse and exploratory. If training stops here, it might seem like RLVR limits the model’s creativity.
However, if training continues long enough, the model enters the “Exploration Stage.” At this point, the probabilities of the previously high-reward tokens start to stabilize because they’re already near their maximum. This opens a window for the model to occasionally sample and discover new, potentially optimal tokens that it previously overlooked due to their low initial probability. When these new, better solutions are found and receive positive feedback, their probabilities increase, and the model’s reasoning capability “expands.” This prolonged training allows the model to genuinely explore and develop novel reasoning strategies.
The researchers illustrate this with a simple example, showing how the probabilities of different actions change over time, confirming their two-stage theory. They also highlight that over-exploitation in the early stage can indeed lead to a narrowing of capabilities, while extended training can foster expansion.
Also Read:
- Adaptive Sampling Enhances Stability and Efficiency in LLM Reinforcement Learning
- Targeted Expert Guidance Enhances LLM Reasoning and Exploration
Prolonging Training for Expansion
Building on these insights, the paper suggests a practical way to encourage this expansion: by focusing policy probability updates exclusively on “relative negative gradients.” In simpler terms, instead of just reinforcing correct answers, the model learns significantly from its mistakes or less optimal choices. This approach, implemented in variants like GRPO-N and GSPO-N, helps maintain the model’s diversity and allows for longer, more stable training.
Experiments using models like Qwen2.5-Math-7B and Llama-3.2-3B-Instruct on various math and reasoning benchmarks showed promising results. The GRPO-N and GSPO-N methods achieved competitive performance while significantly preserving or even increasing the model’s entropy (a measure of diversity and exploratory capacity), unlike standard GRPO which often caused entropy to collapse. This indicates that learning from negative examples helps prevent the model from becoming too narrow in its approach.
A case study further demonstrated this. While a standard GRPO model might repeatedly make the same coding errors, GRPO-N produced fewer errors and showed instances where it initially made a mistake but later refined and corrected its reasoning. This suggests that by carefully managing how models learn from both positive and negative feedback, we can guide them towards more robust and genuinely expansive reasoning abilities.
This research offers a crucial understanding of how RLVR influences LLMs, reconciling conflicting views and providing a theoretical and empirical foundation for developing more advanced reasoning capabilities. It emphasizes the importance of sustained training and smart allocation of learning signals to truly unlock the potential of these powerful AI models. For more details, you can read the full paper here: The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic View.
