TLDR: A new research paper introduces ExGRPO, a framework that enhances large language models’ reasoning abilities by strategically managing and reusing past learning experiences. It identifies rollout correctness and trajectory entropy as key indicators of experience value, prioritizing medium-difficulty problems and low-uncertainty reasoning paths. ExGRPO consistently improves performance and stabilizes training across various models, demonstrating the importance of intelligent experience management in reinforcement learning for reasoning.
Large language models are becoming increasingly adept at complex reasoning tasks, thanks in part to a training method called Reinforcement Learning from Verifiable Rewards (RLVR). This approach allows models to learn by generating chains of thought and receiving feedback on whether their final answer is correct. However, a significant challenge with traditional RLVR methods is their inefficiency: they often discard valuable learning experiences after a single training update.
Imagine a student who solves a math problem, learns from it, and then immediately forgets the entire process, only to start fresh on the next problem. This is similar to how many RLVR algorithms operate, leading to wasted computational resources and missed opportunities for deeper learning. While the concept of reusing past experiences, known as experience replay, has been beneficial in other areas of reinforcement learning, its application to large reasoning models (LRMs) has remained largely unexplored.
A new research paper, EXGRPO: LEARNING TO REASON FROM EXPERIENCE, delves into this gap, investigating what truly makes a reasoning experience valuable for LRMs. The authors, Runzhe Zhan, Yafu Li, Zhi Wang, Xiaoye Qu, Dongrui Liu, Jing Shao, Derek F. Wong, and Yu Cheng, found that two key factors indicate the value of an experience: the correctness of the model’s attempt and the ‘entropy’ of its reasoning path. Essentially, problems of intermediate difficulty and reasoning steps that show less uncertainty (lower entropy) are most beneficial for learning.
Based on these insights, the researchers propose ExGRPO (Experiential Group Relative Policy Optimization). This innovative framework is designed to strategically organize and prioritize valuable experiences. It maintains a ‘replay buffer’ of successful reasoning attempts, categorizing them into ‘buckets’ based on their correctness levels. ExGRPO then employs a smart sampling strategy, prioritizing experiences from the most beneficial buckets – those medium-difficulty problems – and selecting the reasoning paths within them that exhibit the lowest entropy.
During the actual training process, ExGRPO uses a ‘mixed-policy objective.’ This means it balances learning from fresh, new explorations with strategically reusing these carefully selected past experiences. To ensure stable and effective learning, especially in the early stages, ExGRPO also incorporates ‘policy shaping’ to modulate gradients from replayed data and a ‘delayed start mechanism’ that activates experience replay only after the model has achieved a certain level of basic competence.
The experimental results are compelling. Tested across five different language models (ranging from 1.5 billion to 8 billion parameters) on both mathematical and general reasoning benchmarks, ExGRPO consistently improved performance. It showed an average gain of +3.5 points on mathematical benchmarks and +7.6 points on general benchmarks compared to standard on-policy RLVR. Furthermore, ExGRPO proved crucial in stabilizing training for both stronger and weaker models, where traditional on-policy methods often failed or collapsed.
Also Read:
- GRPO-𝜆: Advancing LLM Reasoning Through Enhanced Credit Assignment
- New Method Stabilizes Reinforcement Learning for Efficient LLM Reasoning
These findings underscore the importance of principled experience management as a critical ingredient for making RLVR more efficient and scalable for large reasoning models. By intelligently reusing past learning, ExGRPO helps models learn faster, more effectively, and with greater stability, paving the way for more advanced and reliable AI reasoning capabilities.
