TLDR: This research paper investigates Reinforcement Learning with Verifiable Rewards (RLVR) for large reasoning models. It theoretically and empirically shows that RLVR is constrained by the base model’s existing knowledge, acting as a conservative reweighting mechanism rather than expanding true reasoning capabilities. While RLVR improves precision (pass@1), it often narrows the diversity of solutions (empirical support shrinkage outweighs expansion) and reduces overall answer diversity, even if token-level uncertainty increases. The paper suggests that RLVR primarily refines existing knowledge, highlighting the need for explicit exploration mechanisms to truly extend AI’s reasoning horizons.
Recent advancements in artificial intelligence, particularly with large reasoning models, have brought a method called Reinforcement Learning with Verifiable Rewards (RLVR) into the spotlight. This technique is seen as a promising way to boost AI’s ability to solve complex logical problems. However, a fundamental question has lingered: does RLVR truly expand an AI model’s capacity to reason, or does it simply make the model better at producing answers it already knows, leading to improved accuracy but not necessarily new insights?
A new study titled “The Invisible Leash: Why RLVR May Not Escape Its Origin” delves into this very question, offering fresh perspectives on the potential limitations of RLVR. The researchers, including Fang Wu, Weihao Xuan, Ximing Lu, Zaid Harchaoui, and Yejin Choi, present both theoretical arguments and empirical evidence to shed light on this debate. You can find the full paper here.
The Core Limitation: Support Preservation
One of the study’s key theoretical insights is that RLVR is fundamentally constrained by what the base model already “knows” or can “sample.” Imagine a base AI model as having a certain range of possible solutions it can generate. The study suggests that RLVR cannot teach the model to come up with solutions that had zero initial probability of being generated by the base model. In essence, RLVR acts as a conservative reweighting mechanism, primarily focusing on improving the precision of solutions already within the base model’s existing capabilities, rather than discovering entirely new ones.
This concept is termed “support preservation.” It means that if a correct solution is completely outside the base model’s initial “support” (i.e., it has zero probability of being generated), RLVR cannot “discover” it. This is because RLVR relies on feedback from samples generated by the base model itself. If a solution is never sampled, it can’t receive a reward signal, and thus, the model can’t learn to produce it.
The Entropy-Reward Trade-off
The research also identifies an “entropy-reward trade-off.” While RLVR is effective at reliably enhancing the precision of answers (meaning it gets the right answer more often on the first try, known as pass@1), it may also progressively narrow the model’s exploration. This narrowing can lead to the model overlooking correct solutions that are less common or “underrepresented” in its initial distribution. In simpler terms, by focusing on the most likely correct answers, RLVR might inadvertently reduce the diversity of solutions the model can generate, even if some of those diverse solutions are also correct.
This trade-off is particularly evident when looking at different sampling budgets. For small budgets (like pass@1), RLVR excels because it concentrates probability on the most probable correct answers. However, for larger sampling budgets (like pass@k where k is large), the base model might actually perform better. This is because the base model, with its broader and more diverse distribution, might occasionally sample correct answers from its “long-tail” (less probable but still possible) distribution, which RLVR might have suppressed in its quest for precision.
Empirical Validation: Shrinkage Outweighs Expansion
To validate their theories, the researchers conducted extensive experiments across various domains, including mathematics, logical reasoning, factual QA, and code generation tasks. They introduced the concept of “empirical support” to track how RLVR impacts the set of correct completions a model can generate.
The findings were compelling: while RLVR consistently improved pass@1 accuracy, the “shrinkage” of empirical support generally outweighed its “expansion.” This means that RLVR more frequently failed to recover correct answers that were previously accessible to the base model (shrinkage) than it managed to assign non-negligible probability to previously underrepresented correct completions (expansion). This reinforces the idea that RLVR acts more as a conservative reweighting mechanism, refining existing knowledge rather than fundamentally expanding the model’s reasoning horizons.
Interestingly, the study also observed a decoupling between “token-level entropy” and “answer-level entropy.” Token-level entropy, which measures uncertainty at each step of generation, sometimes increased with RLVR. This might suggest more complex or longer reasoning chains. However, answer-level entropy, which measures the diversity of final answers, consistently declined. This indicates that even if the model seems more “exploratory” at a step-by-step level, its paths ultimately converge onto a smaller set of distinct final answers. This phenomenon is termed “local stochasticity without global exploration.”
Also Read:
- Unpacking Inverse Scaling: How Longer AI Reasoning Can Reduce Accuracy
- Unlocking AI Reasoning: How Language Mixing Enhances Large Language Models
Breaking the Invisible Leash
The study concludes that despite its empirical success in improving precision, RLVR may have inherent limits in truly extending the reasoning capabilities of large language models. To overcome this “invisible leash,” future algorithmic innovations might be necessary. These could include explicit exploration mechanisms or hybrid strategies that intentionally introduce probability mass into underrepresented solution regions. This work provides valuable insights into both the strengths and limitations of current RLVR approaches, guiding the path for developing AI systems that can genuinely unlock new reasoning capacities.
