TLDR: The research paper introduces EDGE-GRPO, a new algorithm designed to overcome the ‘advantage collapse’ problem in Group Relative Policy Optimization (GRPO) for Large Language Models (LLMs). Advantage collapse occurs when sparse rewards lead to identical advantages within response groups, hindering effective training. EDGE-GRPO addresses this with two main components: Guided Error Correction (GEC), which enhances response diversity by strategically correcting incorrect answers, and Entropy-Driven Advantage (EDA), which uses policy entropy to provide more fine-grained and diverse training signals. Experiments show that EDGE-GRPO significantly improves performance on reasoning benchmarks with high data efficiency, using only 1,000 training samples.
Large Language Models (LLMs) have made significant strides in improving their step-by-step reasoning abilities, largely thanks to advancements in reinforcement learning. One notable algorithm in this area is Group Relative Policy Optimization (GRPO), which has gained attention for its efficiency in training by calculating rewards and advantages across groups of sampled responses, rather than relying on a value function like in PPO.
The Challenge: Advantage Collapse
However, GRPO faces a critical issue known as “advantage collapse.” This problem arises because many reward systems used with GRPO are “sparse,” meaning that all responses within a group often receive identical rewards, especially when the final answer is the sole determinant. When all responses in a group are either correct or incorrect, their calculated advantages become zero. This lack of distinction means the model receives no effective signal to learn from, severely limiting the efficiency of the training process.
Existing efforts to combat advantage collapse typically fall into two categories: encouraging the model to reflect and generate more diverse responses, or introducing internal feedback to strengthen the training signal. However, these approaches have their limitations. For instance, forcing models to reflect on incorrect answers often yields minimal improvement, and the relationship between response correctness and the model’s confidence (measured by policy entropy) is often miscalibrated.
Introducing EDGE-GRPO: A Dual-Level Solution
To address these challenges, researchers have proposed a new algorithm called EDGE-GRPO, which stands for Entropy-Driven GRPO with Guided Error Correction. This innovative approach tackles advantage collapse from two key perspectives: at the response level and at the signal level.
Guided Error Correction (GEC) for Response Diversity
At the response level, EDGE-GRPO introduces Guided Error Correction (GEC). The research found that a model’s ability to self-correct through reflection is quite limited, often leading to groups of entirely incorrect responses when facing difficult problems. GEC intervenes by ensuring that each group of responses contains a mix of positive (correct) and negative (incorrect) examples, even for challenging questions. For incorrect responses, GEC applies one of three strategies with predefined probabilities:
- Prompt and Regenerate: The model is given a simple reflection prompt and asked to regenerate its answer, offering a chance for self-correction.
- Direct Answer Injection: The correct answer is directly provided along with a reflection prompt.
- Reference Solution Replacement: The incorrect response is entirely replaced with an external, correct reference solution.
These strategies enhance response diversity, providing more effective training signals and preventing advantages from uniformly collapsing to zero.
Entropy-Driven Advantage (EDA) for Signal Diversity
Even with GEC, there might still be uniform advantages among correct or incorrect responses. To enable finer-grained differentiation, EDGE-GRPO incorporates Entropy-Driven Advantage (EDA) at the signal level. Policy entropy measures the model’s uncertainty over its generated response. The research observed that models often misjudge their confidence: many incorrect responses exhibit low entropy (overconfidence), while many correct responses have high entropy (lack of confidence).
EDA scales the initial advantages based on this policy entropy. It assigns higher advantages to correct and confident responses, while imposing harsher penalties on incorrect but overly confident responses. This mechanism ensures that different responses receive distinct advantage values, further enhancing the model’s ability to learn from its outputs and significantly mitigating the advantage collapse problem.
Also Read:
- New Algorithm Enhances LLM Tool Use with Adaptive Learning
- UloRL: Boosting LLM Reasoning with Efficient Ultra-Long Output Training
Impressive Results with Minimal Data
Extensive experiments on several mathematical reasoning benchmarks demonstrate the effectiveness and superiority of EDGE-GRPO. The method consistently achieves over 20% performance improvement compared to base models. Notably, EDGE-GRPO achieves competitive and excellent performance using only 1,000 training samples, a significantly smaller dataset compared to what many other open-source models require. This highlights the algorithm’s remarkable data efficiency.
The research also shows that EDGE-GRPO maintains a higher level of intra-group advantage variance during training, directly confirming its success in alleviating the advantage collapse problem. This innovative approach offers a promising direction for enhancing the reasoning capabilities of LLMs, especially in complex tasks like mathematics and coding, by providing more robust and diverse training signals. You can find more details about this research at the research paper.
