TLDR: A new research paper investigates entropy dynamics in Large Language Models (LLMs) trained with Reinforcement Learning with Verifiable Rewards (RLVR). It reveals that entropy collapse, a common issue in RLVR, correlates with reduced response diversity and model miscalibration. The study identifies off-policy updates, training data diversity, and clipping thresholds as key factors influencing entropy. Crucially, it demonstrates that tokens with positive advantages drive entropy collapse and proposes ‘Progressive Advantage Reweighting’ to effectively regulate entropy and improve LLM performance.
Large Language Models (LLMs) have shown incredible advancements in complex reasoning tasks like mathematics and coding, largely thanks to a training method called Reinforcement Learning with Verifiable Rewards (RLVR). While RLVR significantly boosts these capabilities, it often leads to a problem known as ‘entropy collapse’. This phenomenon, where the model’s internal uncertainty or diversity of thought decreases drastically, can cause LLMs to settle for less optimal solutions and hinder further improvement.
A recent study, titled Revisiting Entropy in Reinforcement Learning for Large Reasoning Models, by Renren Jin, Pengzhi Gao, Yuqi Ren, Zhuowen Han, Tongxuan Zhang, Wuwei Huang, Wei Liu, Jian Luan, Deyi Xiong, and their colleagues, delves deep into this issue. The researchers conducted extensive experiments to understand how entropy behaves in LLMs trained with RLVR and how it relates to various aspects of model performance, including the diversity of responses, how well the model is calibrated (its confidence in predictions), and its overall effectiveness across different tasks.
Understanding Entropy Collapse
Entropy collapse means that an LLM’s probability distribution over its vocabulary becomes highly concentrated on a small set of tokens. In simpler terms, the model becomes less exploratory and more focused on exploiting known paths, which can lead to premature convergence to a local optimum rather than finding the best possible solution. While previous research has proposed methods to counteract this, a comprehensive understanding of entropy in RLVR has been lacking.
Key Findings on Entropy Dynamics
The study revealed several crucial insights into entropy dynamics:
-
Response Diversity: There’s a strong positive correlation between an LLM’s entropy and the diversity of its generated responses. Models with lower entropy tend to produce less varied outputs.
-
Prompt Entropy: The entropy of LLMs on prompts (the input questions) consistently declines during RLVR training, with a more significant drop for in-domain prompts compared to out-of-domain ones. However, this prompt entropy showed only a weak link to the accuracy of the responses.
-
Performance vs. Entropy: Interestingly, LLM performance can continue to improve even when entropy doesn’t decrease. This suggests that performance gains aren’t solely achieved by sacrificing entropy. In fact, severe entropy collapse can lead to performance degradation, while adaptive entropy regularization can help maintain entropy and improve accuracy.
-
Task-Dependent Correlations: The relationship between entropy and performance varies by task. For instance, coding capabilities showed a strong negative correlation with entropy (lower entropy meant better coding), while mathematical reasoning and instruction following tasks had weaker correlations.
-
Miscalibration: Entropy collapse is linked to model miscalibration, where LLMs become overly confident in their predictions, even incorrect ones. More severe entropy collapse corresponds to stronger miscalibration, a problem that entropy regularization can help alleviate.
Factors Influencing Entropy
The researchers identified three critical factors that influence how entropy changes during RLVR training:
-
Off-Policy Updates: More off-policy updates (when the model’s parameters change after data collection but before all updates are applied) lead to more pronounced entropy collapse. While this can result in higher rewards on the training set, the performance gains on unseen test data are much smaller, suggesting a risk of overfitting.
-
Training Data Diversity: Lower diversity in the training data contributes to more severe entropy collapse. The study also found that the sheer size of the training data isn’t the only determinant of performance; models trained on significantly smaller, carefully selected datasets (e.g., 600 samples) could achieve comparable performance to those trained on much larger ones (e.g., 17,000 samples).
-
Clipping Thresholds: These are parameters in the optimization process that limit how much the model’s probabilities can change. A higher upper clipping threshold helps prevent entropy collapse, while a lower one intensifies it. Similarly, increasing the lower clipping threshold mitigates collapse, and decreasing it amplifies it. Surprisingly, the study found that LLMs could be trained stably even without any clipping, achieving competitive performance.
Regulating Entropy for Better Performance
A significant finding was that tokens with ‘positive advantages’ (those that contribute positively to the reward) are the primary drivers of entropy collapse. When these tokens are updated, their probabilities increase, and since they are often high-probability tokens, this concentrates the probability distribution, reducing entropy. Conversely, tokens with negative advantages, when updated, can counteract this concentration.
Based on this, the researchers proposed a novel method called Progressive Advantage Reweighting (Prog-Adv-Reweight). This approach dynamically adjusts the importance (loss weights) given to tokens with non-negative advantages during training. By carefully controlling these weights, Prog-Adv-Reweight can effectively regulate model entropy. The study demonstrated that this method not only mitigates entropy collapse but also maintains competitive performance across various benchmarks, offering a simple yet effective way to improve LLMs trained with RLVR.
Also Read:
- Enhancing LLM Reasoning: A New Method to Overcome Repetitive Reflections
- Unpacking the Role of Exploration in AI Reasoning: Why Rare Thoughts Matter
Conclusion
This comprehensive investigation sheds light on the complex interplay between entropy and performance in LLMs trained with RLVR. By identifying key factors influencing entropy dynamics and proposing an effective regulation mechanism, the research provides valuable insights for developing more robust and capable large language models in the future.
