TLDR: Reinforcement Learning with Verifiable Rewards (RLVR) paradoxically shrinks Large Language Models’ (LLMs) reasoning capabilities instead of expanding them. This paper identifies two key causes: ‘negative interference,’ where learning to solve some problems reduces the ability to solve others, and the ‘winner-take-all’ phenomenon, where RLVR disproportionately reinforces already high-likelihood solutions, neglecting harder problems and narrowing solution strategies. To combat this, the authors propose SELF (Selective Examples with Low-likelihood and Forward-KL), a data curation algorithm that focuses learning on low-likelihood problems and preserves behavioral diversity, demonstrating improved Pass@k performance.
Reinforcement Learning with Verifiable Rewards (RLVR) has become a popular technique for enhancing the reasoning abilities of Large Language Models (LLMs), particularly in complex tasks like mathematical problem-solving and programming. The core idea behind RLVR is to train LLMs using a simple binary signal: either a solution is objectively correct (reward +1) or it’s not (reward -0.5 or -1), removing the need for extensive human annotations. This approach was believed to foster new reasoning strategies, allowing LLMs to go beyond the capabilities of their initial base models.
However, recent research, including a new paper titled The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models, suggests a surprising paradox: RLVR might actually shrink the reasoning boundary of LLMs instead of expanding it. This means that while LLMs might get better at solving certain problems, they could lose the ability to solve others that they previously could, or become less diverse in their problem-solving approaches.
The paper, authored by Phuc Minh Nguyen, Chinh D. La, Duy M. H. Nguyen, Nitesh V. Chawla, Binh T. Nguyen, and Khoa D. Doan, delves into why this ‘shrinkage’ occurs by analyzing the learning dynamics of RLVR. They identify two key phenomena that explain this counterintuitive outcome.
Negative Interference
The first phenomenon is called ‘negative interference’. In the context of LLMs, each problem can be thought of as inducing its own unique learning environment. The researchers found that when an LLM learns to solve a specific set of training problems using RLVR, it can actively reduce its ability to correctly solve other problems. This leads to a decline in ‘Pass@k’ performance, which measures the probability of generating a correct solution within ‘k’ attempts. Essentially, improving on one area inadvertently harms performance in another.
Winner-Take-All Phenomenon
The second critical finding is the ‘winner-take-all’ phenomenon. This occurs because RLVR, due to its inherent ‘on-policy sampling’ nature, tends to disproportionately reinforce problems that the base model already has a high likelihood of solving correctly. Problems that are initially harder for the base model, or have a low likelihood of correct solutions, are suppressed or neglected. Over time, this causes the LLM to converge on a narrow set of solution strategies, reducing the diversity of its problem-solving behaviors. This effect is exacerbated by negative interference, as the model’s confidence in correct solutions for ‘weaker’ problems degrades.
For example, in the Minerva benchmark, LLMs often employ both code-based and natural language reasoning. The study observed that during RLVR training, the model progressively collapsed into using only natural language reasoning, even if code reasoning initially offered better accuracy for certain problems. This ‘winner-take-all’ effect meant that the more successful natural language approach dominated, leading to a loss of diversity and reduced performance on problems that benefited from code reasoning.
Also Read:
- DIVER: A New Approach to Enhance LLM Reasoning Through Diverse Exploration
- Targeted Exploration for Enhanced Reasoning in AI Models
The Proposed Solution: SELF
To address these issues, the researchers propose a new data curation algorithm called SELF (Selective Examples with Low-likelihood and Forward-KL). This algorithm is designed to focus RLVR learning specifically on problems where the model’s initial ‘greedy’ response (its most confident answer) fails. By excluding problems that are already easily solvable, SELF prevents them from monopolizing the learning signal.
Additionally, SELF replaces the standard ‘Reverse KL’ regularization with a ‘Forward KL’ objective. This change helps to penalize the model if it starts to ‘forget’ previously learned behaviors, thereby preserving the diversity of its reasoning strategies. Empirical evaluations show that SELF not only improves sample efficiency but also effectively mitigates the coverage shrinkage problem, leading to better Pass@k performance across various mathematical reasoning benchmarks, especially for larger ‘k’ values.
In conclusion, while RLVR is a powerful tool, this research highlights its limitations and offers a new perspective on how to refine it. By understanding and addressing negative interference and the winner-take-all effect, techniques like SELF can help LLMs truly expand their reasoning boundaries rather than inadvertently constraining them.
