TLDR: SIREN (SelectIve entRopy rEgularizatioN) is a novel method designed to overcome entropy collapse and premature convergence in large reasoning models (LRMs) trained with reinforcement learning and verifiable rewards (RLVR). It addresses the limitations of naive entropy regularization, which struggles with LRMs’ vast action spaces and long reasoning trajectories. SIREN employs a two-step entropy masking mechanism—a top-p mask to focus exploration on meaningful actions and a peak-entropy mask to target critical tokens along reasoning paths. Additionally, it uses self-anchored regularization to stabilize training and maintain appropriate entropy levels. This approach significantly improves reasoning performance on mathematical benchmarks, fosters greater response diversity, and ensures sustained exploration, leading to more robust and accurate AI models.
Large Reasoning Models (LRMs) are at the forefront of artificial intelligence, showing incredible potential in complex tasks like mathematics and coding. These models often learn and improve through a technique called Reinforcement Learning with Verifiable Rewards (RLVR). While powerful, RLVR faces a significant hurdle: a problem known as ‘entropy collapse’ and ‘premature convergence’. This means the AI model can become too rigid too quickly, leading to repetitive and less diverse outputs, which ultimately limits its ability to explore new solutions and improve its performance.
Traditionally, a common way to encourage exploration in reinforcement learning is ‘naive entropy regularization’. This method aims to keep the model’s predictions diverse. However, in the context of LRMs, this approach often falls short. Researchers have found that simply applying naive entropy regularization can either have little effect or, if applied too strongly, cause a ‘global entropy explosion’. This explosion makes the model explore everything indiscriminately, including meaningless actions and states, leading to incoherent and unhelpful outputs.
The core issue stems from two unique characteristics of LRMs: their incredibly vast ‘action space’ (the huge number of possible tokens or words they can choose from) and their ‘long trajectories’ (the extended sequences of reasoning steps they generate). When naive entropy regularization is applied, the model tries to flatten its probability distribution across the entire vocabulary. This leads to disproportionately large entropy gains from diffusing probability mass across many low-utility tokens, causing the model to prioritize increasing entropy over generating meaningful content. Similarly, along long reasoning paths, uncertainty can accumulate and propagate, leading to an explosion of entropy across almost all token positions.
To tackle these challenges, a new method called SIREN (SelectIve entRopy rEgularizatioN) has been proposed. SIREN is designed to confine exploration to a more meaningful and effective subset of actions and states, preventing the pitfalls of uncontrolled exploration. It achieves this through a clever two-step entropy masking mechanism.
The first step is a ‘top-p mask’. This mask limits exploration to what the researchers call the ‘policy nucleus’ – a subset of the vocabulary containing the most semantically important tokens. These are the tokens that are reasonable to explore and are unlikely to lead to incoherent responses. By focusing on this nucleus, SIREN ensures that the model explores within a relevant and high-quality action space, rather than wasting effort on meaningless tokens.
The second step is a ‘peak-entropy mask’. For long reasoning trajectories, not all tokens are equally important for exploration. Some tokens play critical logical roles, guiding the reasoning process, while others are more like semantic fillers. The peak-entropy mask identifies these critical tokens, which tend to exhibit higher entropy, and concentrates regularization efforts on them. This targeted intervention helps prevent the cascade of uncertainty that can lead to entropy explosions along long sequences.
In addition to these masking mechanisms, SIREN introduces ‘self-anchored regularization’. Instead of simply trying to maximize entropy, which can lead to extremes, this approach aims to keep the policy’s entropy at an appropriate, stable level. It does this by measuring the mean squared error between the current aggregated entropy and an ‘entropy anchor’, which is initialized from the model’s inherent uncertainty at the very beginning of training. This self-anchored approach stabilizes training and ensures sustained, effective exploration.
Also Read:
- DIVER: A New Approach to Enhance LLM Reasoning Through Diverse Exploration
- DeepSearch: Enhancing Language Model Reasoning Through Integrated Tree Search Training
The effectiveness of SIREN has been demonstrated across five mathematical benchmarks. For instance, it achieved a significant +6.6 maj@k improvement on AIME24/25 with the Qwen2.5-Math-7B model, outperforming previous entropy-related RLVR approaches. Further analysis confirmed that SIREN promotes greater response diversity and maintains entropy at an appropriate level throughout training, effectively mitigating the premature convergence problem common in RLVR for LRMs. This research highlights the importance of targeted and effective exploration for enhancing the reasoning capabilities of large AI models. You can read the full paper here: Rethinking Entropy Regularization in Large Reasoning Models.
