TLDR: A new research paper introduces STEER, a method to prevent “entropy collapse” in Large Language Models (LLMs) trained with Reinforcement Learning (RLVR). Entropy collapse means LLMs lose diversity in their responses, hindering reasoning. STEER analyzes and controls token-level entropy changes directly, unlike previous indirect methods. By adaptively reweighting tokens to keep entropy changes moderate, STEER significantly improves LLM performance and training stability on mathematical reasoning tasks.
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a powerful technique for enhancing the reasoning abilities of Large Language Models (LLMs). However, a significant challenge in this training process is ‘entropy collapse.’ This phenomenon refers to a rapid loss of diversity in the model’s responses, which can severely limit its ability to generalize and explore different solutions.
Existing methods designed to prevent entropy collapse often fall short because they try to control entropy indirectly. These approaches might adjust factors like the advantage signal or generation probability, but their effectiveness is limited, and they can sometimes fail or even cause unintended side effects, such as amplifying entropy fluctuations.
A new research paper, Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective, introduces a novel way to understand and address this problem. The authors, Zhezheng Hao, Hong Wang, Haoyang Liu, Jian Luo, Jiarui Yu, Hande Dong, Qiang Lin, Can Wang, and Jiawei Chen, propose that the overall entropy dynamics during training are a result of the accumulation of per-token entropy changes. By analyzing these changes at a fine-grained, token level, they can gain a clearer insight into how entropy evolves.
Understanding Entropy Dynamics
The researchers developed a quantitative analysis framework that identifies the direct factors governing token-level entropy change. This framework introduces a ‘token-level entropy change indicator’ and divides the policy update space into four quadrants:
- Quadrant I (Exploitation): For high-probability correct tokens, rewarding these behaviors decreases entropy.
- Quadrant II (Exploration): For low-probability correct tokens, rewarding these rare-but-correct behaviors increases entropy.
- Quadrant III (Suppression): For low-probability incorrect tokens, penalizing errors decreases entropy.
- Quadrant IV (Error-Correction): For high-probability incorrect tokens, penalizing overconfident errors increases entropy by encouraging the model to seek alternatives.
This framework helps explain why entropy collapse occurs: the exploitation-driven, entropy-decreasing updates consistently overpower the exploration-driven, entropy-increasing updates. It also clarifies the asymmetric effects of PPO-style ratio clipping (where different clipping thresholds can either increase or decrease entropy) and the impact of positive versus negative sample weighting (training on negative samples tends to maintain higher entropy, while positive samples can lead to rapid collapse).
Crucially, the paper highlights a peril in targeting high-entropy tokens directly. While it might seem intuitive to focus on uncertain states to promote exploration, the analysis shows that high-entropy tokens are sources of *fluctuation*, not reliable entropy increase. Amplifying updates on these tokens can inadvertently accelerate entropy collapse rather than stabilizing it.
Introducing STEER
To overcome the limitations of existing methods, the authors introduce Stabilizing Token-level Entropy-changEviaReweighting (STEER). This adaptive, fine-grained reweighting scheme aims to keep the stepwise entropy change within a moderate band. Unlike previous approaches that only consider a subset of factors, STEER takes a comprehensive approach by considering all relevant factors influencing entropy change.
STEER works by applying an exponential-decay mapping to token weights. Tokens that exhibit excessively large entropy changes receive substantially reduced weights, ensuring that potentially disruptive updates are suppressed. This mechanism stabilizes training without impeding the model’s overall learning, as most tokens retain weights close to one.
Experimental Validation
Extensive experiments were conducted on various mathematical reasoning benchmarks, including AIME24, AIME25, AMC23, MATH-500, Minerva Math, and OlympiadBench. STEER consistently outperformed classical RLVR baselines and existing entropy intervention methods, achieving stronger downstream performance. For instance, STEER improved average performance by 2.7 points over the second-best method (OPO) and 3.4 points over the third-best (Clip-Cov) on the Qwen2.5-Math-7B model.
Furthermore, STEER demonstrated superior entropy control, maintaining stable entropy dynamics even in extreme training scenarios where other methods failed, leading to either entropy collapse or an ‘entropy explosion’ (excessive uncertainty). The ablation studies confirmed the necessity of continuous token-level reweighting for precise entropy control.
Also Read:
- Dynamic Exploration for LLMs: Adaptive Entropy Regularization Improves Reasoning
- Enhancing Language Model Reasoning Through Representation-Based Exploration
Conclusion
The research provides a new perspective on entropy interventions in RLVR by focusing on token-level entropy change. STEER offers a practical and effective solution for developing robust and stable LLM training algorithms, significantly mitigating entropy collapse and enhancing exploration. This work not only advances our understanding of RL dynamics but also paves the way for more reliable and powerful LLMs for complex reasoning tasks.
