TLDR: A new research paper introduces the Verifiable Stepwise Reward Mechanism (VSRM), a rule-based system that helps large reasoning models (LRMs) become more efficient. VSRM rewards effective intermediate steps and penalizes ineffective ones, significantly reducing the models’ tendency to “overthink” and generate excessively long outputs, all while maintaining or improving their accuracy on complex tasks.
Large reasoning models (LRMs) have made significant strides in tackling complex reasoning tasks, especially when enhanced by reinforcement learning techniques that use verifiable rewards. These models, like OpenAI’s O1 and DeepSeek-R1, can break down problems into steps and explore multiple solutions, leading to breakthroughs in areas like coding and mathematics.
However, a major challenge for LRMs is “overthinking.” This happens when models spend too much computational effort on simple problems, generating excessively long responses. This not only slows them down but can also sometimes lead to incorrect answers. The core issue often lies in their training, where a binary reward system encourages models to generate a lot of content to ensure correctness, even if it means being redundant.
Current approaches to make LRMs more efficient often involve setting limits on output length or adapting reasoning modes based on problem difficulty. While these methods offer some relief, they require accurately predicting task complexity beforehand, which can limit their flexibility and reliability.
A new research paper, titled “Promoting Efficient Reasoning with Verifiable Stepwise Reward,” by Chuhuai Yue, Chengqi Dong, Yinan Gao, and their colleagues from Meituan, Fudan University, and other institutions, re-examines this overthinking problem. They propose that the key to solving it is to encourage steps that genuinely improve accuracy while penalizing those that are ineffective or meaningless. You can read the full paper here: Promoting Efficient Reasoning with Verifiable Stepwise Reward.
Introducing the Verifiable Stepwise Reward Mechanism (VSRM)
To achieve this, the researchers developed a novel rule-based system called the Verifiable Stepwise Reward Mechanism (VSRM). Unlike traditional methods that give a reward only at the very end of a reasoning process, VSRM provides timely feedback at intermediate stages. This approach is intuitive because reasoning naturally unfolds step-by-step.
VSRM avoids the complexities and instabilities of training a separate “process reward model.” Instead, it uses a rule-based calculation for rewards, making it more interpretable. Here’s a simplified look at how it works:
-
Step Separation: The system first breaks down the model’s entire reasoning output into smaller segments, or “sub-rollouts.” It identifies natural breaking points using special tokens like “however,” “thus,” or “wait,” which often signal a transition to a new reasoning step.
-
Intermediate Reward Assignment: For each sub-rollout, VSRM treats it as a new mini-problem and generates multiple candidate answers. The reward for that intermediate step is then based on the average accuracy of these candidate answers.
-
Difference-Based Rewards: To effectively penalize ineffective steps, VSRM assigns rewards based on the difference in average accuracy between consecutive steps. If a step improves accuracy, it’s rewarded; otherwise, it’s penalized or suppressed.
-
Propagation Decay: To prevent sparse reward signals (where many steps might have zero reward), VSRM incorporates a “propagation decay” mechanism. If a step doesn’t show an immediate significant change in accuracy, the system looks ahead a few steps. If a significant change occurs later, a discounted reward is propagated back to the earlier steps, helping the model learn more efficiently.
This VSRM can be seamlessly integrated with popular reinforcement learning algorithms like PPO and Reinforce++, making it practical for training large reasoning models.
Experimental Validation and Impact
The researchers conducted extensive experiments on standard mathematical reasoning benchmarks, including AIME24, AIME25, MATH-500, AMC23, Minerva, and OlympiadBench. They tested VSRM with various base models, such as DS-Distill-1.5B, DS-Distill-7B, and DeepScaleR.
The results were compelling: VSRM achieved a substantial reduction in output length (token consumption) while maintaining or even slightly improving the original reasoning performance of the models. This demonstrates an optimal balance between efficiency and accuracy. When compared to other existing methods, VSRM consistently showed superior performance in both reducing output length and preserving reasoning capabilities.
Further analysis revealed that VSRM effectively suppresses “ineffective steps”—those parts of the reasoning process that don’t contribute to the final accuracy. It also encourages “effective reasoning,” meaning the model still explores valuable problem-solving paths without getting stuck in redundant loops. This was evident in the reduced frequency of detected overthinking and the sustained pass@k scores, which measure a model’s ability to find correct answers across multiple attempts.
Also Read:
- Teaching LLMs to Be Concise: A New Approach to Efficient Reasoning
- Enhancing LLM Accuracy in Complex Reasoning Tasks
Conclusion
The Verifiable Stepwise Reward Mechanism offers a fundamental solution to the overthinking problem in large reasoning models. By focusing on rewarding effective intermediate steps and penalizing ineffective ones, VSRM enables LRMs to reason more efficiently without compromising their powerful problem-solving abilities. This work paves the way for more practical and deployable large reasoning models in real-world applications.
