TLDR: The paper addresses the “zero-reward barrier” in Reinforcement Learning (RL) for Large Language Models (LLMs) where learning stalls if the model never samples correct solutions. It shows that common algorithmic improvements like dense rewards or diversity incentives fail in this scenario. Instead, a simple data-centric intervention—adding easier training samples—enables the model to eventually solve hard tasks, even from a zero-reward start, without modifying the RL algorithm itself. The key is to include samples of appropriate difficulty, or simply mix all available samples of varying difficulty, to facilitate skill transfer.
Reinforcement Learning (RL) has become a vital technique for enhancing large language models (LLMs) in complex reasoning tasks, such as solving mathematical problems or navigating the web. However, its effectiveness often hinges on the base model occasionally generating correct solutions. When an LLM consistently fails to produce any correct answers, RL training hits a significant roadblock known as the “zero-reward barrier.” In this scenario, the model receives no rewards, leading to zero gradients, which means its parameters remain unchanged, and no learning occurs.
A recent research paper, titled “WHAT CAN YOU DO WHEN YOU HAVE ZERO REWARDS DURING RL?” by Jatin Prakash and Anirudh Buvanesh, delves into this critical problem. The authors investigate what strategies can be employed when LLMs face this zero-reward challenge during RL post-training.
The Challenge: Zero Rewards in Action
To study this, the researchers used a simplified yet challenging task: finding a path from a source to a destination in a star graph, a problem introduced in earlier work. This task allowed for controlled experiments with varying difficulty levels. Specifically, they focused on a difficult variant, the “Degree-10-Path-10” graph, where the base LLM initially had a zero success rate.
The paper evaluated several state-of-the-art methods designed to address sparse rewards in RL. These included approaches that incorporate dense rewards (like VinePPO and Rewarding Progress), improve credit assignment for intermediate steps, and incentivize diverse responses (such as Best-of-N aware finetuning). The surprising finding was that all these methods failed to overcome the zero-reward barrier on the hard graph search task. Despite their sophisticated designs, they couldn’t kickstart learning when the model never produced a correct answer.
Why Did Baselines Fail?
The authors conducted a detailed analysis of these failures. For methods relying on “dense rewards” (VinePPO and Rewarding Progress), the problem was that these rewards only become non-zero if some rollouts under the current or a “prover” policy succeed. If the base model never finds a correct path, these step-level advantages also remain zero, providing no learning signal. Instantiating an effective “prover” for Rewarding Progress also proved challenging, as it needs to be neither too strong nor too weak, and well-aligned with the policy being optimized. Best-of-N aware finetuning, which aims to promote diversity, suffered from unstable training due to very high negative gradients when the failure probability was high, leading to degenerate model responses.
A Simple Yet Effective Solution: Data-Centric Intervention
In stark contrast to the failure of algorithmic improvements, the researchers found that a simple data-centric intervention proved highly effective: adding easier samples to the training dataset. By mixing samples from a simpler variant of the task (e.g., “Degree-5-Path-5” graphs) with the original hard “Degree-10-Path-10” task, the model gradually learned to solve the harder task using only outcome rewards. Crucially, this was achieved without any modifications to the RL algorithm itself.
The study further explored whether all “easy” samples are equally effective. It turns out they are not; adding very easy samples (like “Degree-2-Path-5” or “Degree-5-Path-2”) did not help solve the harder task. The key is to include samples of the “right difficulty” that encourage behaviors transferable to the target task. However, the paper offers an even more practical recipe: instead of trying to pinpoint the exact “right difficulty,” simply mix all available samples of varying difficulty into the training dataset. The model, when trained with naive RL, still learns to solve the hard task, suggesting it implicitly learns the necessary behaviors from the appropriately difficult samples within the mixture.
Also Read:
- Adaptive Sampling Enhances Stability and Efficiency in LLM Reinforcement Learning
- Optimizing LLM Reasoning: The Critical Role of Training Data and Test-Time Compute
Why This Works: Skill Learning and Practical Implications
The authors hypothesize that this approach connects to “skill learning.” Easier samples allow the model to acquire fundamental skills or “correlated actions” from outcome rewards. These learned skills then transfer to more difficult tasks, simplifying the search problem for RL by effectively reducing the action space. For instance, skills like traversing a branch without hallucinating or systematically exploring branches can be learned from easier examples and applied to harder ones.
This research highlights the profound impact of data-centric strategies in RL for reasoning. It suggests that when LLMs face a cold-start scenario with zero success rates, algorithmic tweaks alone are insufficient. Instead, providing a foothold through easier instances enables the model to bootstrap its learning towards more challenging problems. This finding offers a practical guide for RL practitioners and emphasizes the need for future evaluations to include settings where base models initially struggle, providing a more robust measure of progress in exploration and reasoning.
For more details, you can read the full research paper here: WHAT CAN YOU DO WHEN YOU HAVE ZERO REWARDS DURING RL?
