TLDR: A new study reveals that the GRPO reinforcement learning algorithm causes language models to make overconfident probability predictions for uncertain outcomes, unlike PPO and RLOO which yield well-calibrated models. This overconfidence stems from a ‘group standard normalization’ step in GRPO’s advantage estimation, which can be fixed by removing it. The findings, validated on synthetic and biological data, emphasize the need for unbiased policy gradients in training AI for reasoning about stochastic events.
Reinforcement learning (RL) has shown remarkable success in enhancing the accuracy of language models, particularly in areas where answers are clear-cut and verifiable, such as mathematics and coding. However, a new study delves into whether these same RL methods are equally effective when dealing with verifiable yet uncertain, or ‘stochastic,’ outcomes, like those encountered in scientific experiments.
The research, titled Uncalibrated Reasoning: GRPO Induces Overconfidence for Stochastic Outcomes, investigates three popular RL algorithms: Group Relative Policy Optimization (GRPO), Proximal Policy Optimization (PPO), and REINFORCE Leave-One-Out (RLOO). These algorithms are commonly used to train language models to generate step-by-step reasoning before providing an answer.
The core finding of the paper is significant: GRPO, when applied to tasks involving binary stochastic outcomes (like predicting the probability of a scientific experiment succeeding or failing), consistently leads to language models that make overly confident probability predictions. In simpler terms, GRPO-trained models tend to be too sure of themselves, even when the underlying outcome is inherently uncertain.
In contrast, PPO and RLOO were found to produce ‘well-calibrated’ models. A well-calibrated model’s predicted probabilities accurately reflect the true likelihood of an event. For example, if a well-calibrated model predicts a 70% chance of an outcome, that outcome should occur approximately 70% of the time in reality.
Why GRPO Becomes Overconfident
The researchers identified the root cause of GRPO’s overconfidence: a component called ‘group standard normalization’ within its advantage estimation process. Advantage estimation is a crucial part of RL, guiding the model to learn which actions lead to better rewards. The study demonstrates that this normalization step causes GRPO to overestimate the ‘advantage’ of predictions that are already overconfident. This creates a problematic positive feedback loop, pushing the model towards increasingly extreme and overconfident probability estimates.
Crucially, the paper shows that simply removing this group standard normalization from GRPO effectively fixes its miscalibration, leading to well-calibrated predictions similar to PPO and RLOO. This highlights the importance of ‘unbiasedness’ as a key design principle for policy gradient methods in RL, especially when training models for reasoning.
Also Read:
- Adaptive Guidance Boosts Reasoning in Small Language Models
- Unpacking Prompt Sensitivity: A Deep Dive into LLM Robustness
Experimental Validation
The findings were supported by experiments conducted on both synthetic data and real-world biological experiments. In the synthetic data tests, GRPO consistently predicted probabilities at the extreme ends (e.g., 1% or 99%) even when the true probabilities were in the middle. PPO, RLOO, and the modified GRPO (without standard normalization) accurately reflected the true probabilities.
A real-world application involved optimizing a large language model (Qwen3-4B) to predict outcomes in CRISPR perturb-seq experiments, which involve perturbing genes and observing their effects on cell phenotypes. Consistent with the synthetic data, GRPO-trained models were highly overconfident, while the other algorithms yielded well-calibrated predictions. Interestingly, all algorithms performed similarly in terms of basic accuracy (whether the prediction was simply right or wrong when thresholded at 0.5), indicating that overconfidence doesn’t necessarily mean lower raw accuracy, but rather a misrepresentation of uncertainty.
This research provides valuable insights for the development of more reliable and trustworthy AI systems, particularly as RL is increasingly applied to complex, real-world problems where understanding and quantifying uncertainty is paramount, such as in scientific discovery and high-stakes decision-making.
