TLDR: CURE is a two-stage framework for training large language models (LLMs) using Reinforcement Learning with Verified Reward (RLVR). It tackles “entropy collapse,” a problem where LLMs become too predictable and stop exploring during training. CURE’s first stage dynamically expands the training data by re-generating responses from “critical tokens” (points of high uncertainty), promoting exploration. The second stage then consolidates these gains through standard training, leading to significant improvements in math reasoning tasks and maintaining diverse model behavior.
Large Language Models (LLMs) have made incredible strides in complex tasks like mathematical reasoning, thanks in part to advancements in Reinforcement Learning with Verified Reward (RLVR). RLVR allows LLMs to learn by receiving precise feedback, enabling them to improve their reasoning capabilities. However, a significant challenge known as “entropy collapse” has hindered further progress in these models.
Entropy collapse occurs when LLMs, during their training, become overly deterministic and predictable. This happens because they repeatedly sample from a fixed set of initial states, leading to a lack of diversity in their behavior. Essentially, the model stops exploring new possibilities, and its performance plateaus, preventing sustained improvements over long training periods.
To address this critical issue, researchers have introduced a novel framework called CURE: Critical-Token-Guided Re-concatenation for Entropy-collapse Prevention. CURE is a two-stage approach designed to strike a crucial balance between exploration (trying new things) and exploitation (refining what works well).
Stage 1: Fostering Exploration
In the first stage, CURE deliberately guides the model towards novel yet coherent contexts. It does this by identifying what are called “critical tokens.” These are points in the model’s generated responses where its internal uncertainty, or “entropy,” is highest. Think of these as decision points where the model is most unsure about what to say next. CURE then takes the part of the response immediately preceding these critical tokens and uses it to create new, refined prompts. The model is then queried again with these new prompts, generating fresh responses. These newly generated responses are combined with the original ones and optimized together. This process encourages the model to explore different reasoning paths and maintain a high level of diversity in its outputs, effectively preventing early entropy collapse.
Stage 2: Consolidating Gains
After the exploration-focused first stage, CURE transitions into its second stage, which emphasizes exploitation. Here, the training reverts to a more traditional method, similar to DAPO (Dynamic Advantage Policy Optimization), using static initial-state sampling. This intentionally places the model in familiar contexts, allowing it to consolidate the knowledge and diverse behaviors discovered during the exploratory first stage. While the dynamic re-concatenation is paused, the model continues to benefit from the enriched policy distribution developed earlier. This phase helps to sharpen the model’s decision boundaries and stabilize its convergence, translating the benefits of exploration into concrete performance gains.
Also Read:
- Boosting LLM Reasoning: A New Approach to Self-Optimization with Entropy
- A New Approach to Harmonize Language Model Training: Dynamic Weighting for Supervised Fine-Tuning and Reinforcement Learning
Key Advantages and Performance
CURE offers several significant advantages. It’s designed to be easily integrated into existing RLVR pipelines, requiring minimal engineering changes. By dynamically re-prompting based on high-uncertainty prefixes, it ensures sustained, high-entropy exploration, continuously steering the model into new but relevant areas of its state space. Furthermore, CURE efficiently converts this exploration into improved performance, demonstrating a strong positive link between policy diversity and higher rewards.
Extensive experiments conducted on the Qwen-2.5-Math-7B model, using the DAPO-Math-17K dataset, have shown impressive results. CURE achieved state-of-the-art performance across six challenging math benchmarks, including AIME24, AIME25, and AMC23. The first stage of CURE alone surpassed other methods, and the second stage further boosted the average accuracy to 54.3%, representing a substantial improvement over the base model. Crucially, CURE consistently maintained higher policy entropy compared to other RLVR approaches, confirming its ability to sustain exploration while achieving superior accuracy.
The research also included ablation studies, which confirmed that the strategic selection of high-entropy critical tokens is vital for CURE’s effectiveness, outperforming random token selection. This highlights that the location of intervention is key to expanding the model’s exploration space meaningfully.
In conclusion, CURE provides a robust solution to the problem of entropy collapse in RLVR-based LLM reasoning. By intelligently combining targeted exploration with stable exploitation, it enables LLMs to achieve higher accuracy in complex tasks like math reasoning while maintaining the necessary diversity for continued improvement. This framework opens new avenues for developing more capable and adaptable large language models. You can find the full research paper here: CURE Research Paper.
