TLDR: A new research paper introduces DARS-B, a novel approach that significantly enhances Large Language Model (LLM) reasoning by optimizing two critical dimensions in Reinforcement Learning with Verifiable Reward (RLVR): ‘Depth’ (the ability to solve harder problems) and ‘Breadth’ (the quantity of training instances per iteration). DARS, a component of DARS-B, adaptively re-weights difficult problems to improve Pass@K performance, while large-breadth training boosts Pass@1 and sustains exploration. The study demonstrates that combining these two dimensions in DARS-B leads to simultaneous gains in both Pass@K and Pass@1, confirming their synergistic relationship in unlocking LLM reasoning capabilities.
Reinforcement Learning with Verifiable Reward (RLVR) is a powerful method that helps large language models (LLMs) improve their reasoning abilities. Unlike traditional methods that rely on human-labeled data or complex reward models, RLVR automatically checks if an LLM’s output is correct, for example, by verifying a mathematical answer or passing code tests. This makes it a promising path for LLMs to continuously learn and evolve on their own.
However, current RLVR frameworks haven’t fully explored two crucial aspects: ‘Depth’ and ‘Breadth’. Depth refers to the hardest problems an LLM can learn to solve, while Breadth is about the number of training examples processed in a single iteration.
Addressing the Depth Challenge
Researchers found that a popular RLVR algorithm called GRPO (Group-based Reinforcement Policy Optimization) has a systematic flaw. It tends to focus too much on problems of medium difficulty, neglecting the really tough ones that are essential for pushing the boundaries of reasoning. This ‘depth neglect’ limits how well LLMs can perform on complex tasks, measured by a metric called Pass@K (which checks if at least one out of K generated solutions is correct).
To fix this, the paper introduces Difficulty Adaptive Rollout Sampling (DARS). DARS works in two stages: first, it quickly estimates how difficult each problem is. Then, it allocates more computational resources to the harder problems through targeted multi-stage rollouts. This re-weights the importance of difficult samples, making it easier for LLMs to learn from them and significantly improving Pass@K performance without adding extra inference cost once the model has converged.
Leveraging Breadth for Performance
The study also highlights the significant impact of ‘Breadth’—the quantity of instances used in each training iteration—on LLM performance, particularly for Pass@1 (which checks if the very first generated solution is correct). Conventionally, batch sizes are relatively small. By drastically increasing the batch size and replacing standard mini-batch updates with full-batch updates over multiple training epochs, the researchers observed a dramatic improvement in Pass@1 performance.
This large-breadth training also helps LLMs maintain high ‘token-level entropy’, which indicates continued exploration and reduces noise in the training process, preventing the model from converging too early.
Also Read:
- How Neuro-Symbolic AI Boosts Reasoning in Language Models
- Adaptive Guidance Boosts Reasoning in Small Language Models
The Synergy of Depth and Breadth: DARS-B
The most exciting finding is the complementary nature of Depth and Breadth. The paper presents DARS-B, which combines the Difficulty Adaptive Rollout Sampling (DARS) with large-breadth training. This integrated approach demonstrates simultaneous improvements in both Pass@K and Pass@1. This confirms that depth (adaptive exploration of difficult problems) and breadth (scaling the quantity of training data) are orthogonal and synergistic dimensions in RLVR, both crucial for unleashing the full reasoning power of LLMs.
The research, detailed in their paper available at arXiv:2508.13755, used Qwen2.5-Math series language models and various mathematical reasoning benchmarks to validate their findings. Their work provides a new perspective on optimizing RLVR pipelines, suggesting that by strategically managing both the difficulty of problems and the volume of training data, we can significantly enhance the reasoning capabilities of large language models.
