TLDR: A new research paper introduces Risk-Sensitive Reinforcement Learning (RS-GRPO) to address the ‘exploration dilemma’ in Large Language Models (LLMs). By adopting a risk-seeking objective, RS-GRPO encourages LLMs to explore more diverse reasoning strategies, overcoming the tendency of standard RL methods to get stuck in narrow solution sets. Experiments on mathematical reasoning benchmarks show that RS-GRPO consistently improves multi-solution performance (pass@k) while maintaining or enhancing single-solution accuracy (pass@1), leading to the discovery of novel reasoning paths.
Large Language Models (LLMs) have shown remarkable capabilities in complex reasoning tasks, especially when enhanced with Reinforcement Learning with Verifiable Rewards (RLVR). However, a significant challenge, termed the ‘exploration dilemma,’ has limited their full potential. This dilemma arises because pre-trained LLMs often start with very specific, or ‘sharply peaked,’ initial policies. This means they tend to stick to a narrow set of solutions, which can improve accuracy for a single best answer (known as pass@1) but severely restricts the diversity of solutions and overall performance on tasks requiring multiple correct answers (pass@k).
Essentially, existing RL methods for LLMs often end up refining what the model already knows rather than helping it discover genuinely new reasoning strategies. This prevents LLMs from expanding their problem-solving capabilities and can lead to stagnation or even a decrease in performance on more general metrics like pass@k.
Introducing Risk-Sensitive Reinforcement Learning
To tackle this exploration dilemma, researchers from Tsinghua University, ETH Zurich, and ByteDance Seed have introduced a novel framework: Risk-Sensitive Reinforcement Learning. Their approach shifts from the standard ‘risk-neutral’ objective, which aims to maximize the average reward, to a ‘risk-seeking’ objective. This new objective intelligently balances between optimizing for the average reward and striving for the maximum possible reward.
This framework leads to a new algorithm called Risk-Sensitive GRPO (RS-GRPO). What’s remarkable about RS-GRPO is its simplicity; it requires only minor code adjustments to existing RL pipelines. By amplifying learning from prompts that the model finds particularly challenging, RS-GRPO encourages deeper exploration of the solution space.
How RS-GRPO Works
The core of RS-GRPO’s effectiveness lies in its ‘risk-sensitive advantage function.’ Unlike standard policy gradients where the advantage is linearly related to the reward, RS-GRPO’s advantage function dynamically re-weights the optimization process. As the ‘risk-sensitivity’ parameter (beta, β) increases, the algorithm places greater emphasis on high-reward outcomes. This means it prioritizes learning from difficult problems where the model initially performs poorly, pushing the policy to explore previously uncharted reasoning paths.
The researchers provided both empirical and theoretical evidence for their claims. A bandit experiment, where a policy was initialized on a suboptimal solution, clearly showed that standard RL methods got trapped in this local optimum. In contrast, risk-sensitive policies with sufficient risk-seeking behavior successfully escaped and converged to the globally optimal reward. Theoretical analysis further supports that the risk-sensitive policy gradient guarantees an improvement for optimal actions when beta is sufficiently large.
Also Read:
- CLPO: A Self-Evolving Learning Approach for Enhanced LLM Reasoning
- Unlocking New Abilities: How Reinforcement Learning Helps Language Models Compose Skills
Impressive Results on Mathematical Reasoning
The RS-GRPO algorithm was rigorously tested on six mathematical reasoning benchmarks, including MATH500, AIME24, AIME25, HMMT-Feb24, HMMT-Feb25, and CMIMC25, using five different LLMs (Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, Qwen2.5-7B, Qwen3-4B-Base, and Llama3.1-8B-Instruct). The results were consistently positive: RS-GRPO significantly improved pass@k performance across the board. Crucially, it achieved these gains while either maintaining or even enhancing pass@1 accuracy, striking a much better balance than previous methods.
For instance, on several models, the standard GRPO algorithm actually performed worse than the base model for high pass@k values, indicating it merely sharpened existing biases. RS-GRPO, however, consistently surpassed the base model, demonstrating its ability to genuinely expand the model’s exploratory boundaries. The analysis also revealed that RS-GRPO leads to a significant increase in the number of unique solutions found, confirming its ability to foster diversity in reasoning paths.
The choice of the risk-sensitivity parameter β is important. An ablation study showed that while larger β values generally improve the solve rate on training data, a moderate β (e.g., β=2) offers an effective trade-off, achieving strong pass@k performance while also enhancing pass@1. This work represents a significant step forward in fine-tuning LLMs, enabling them to discover novel reasoning strategies and overcome the limitations of traditional reinforcement learning approaches. For more details, you can read the full research paper here.
