TLDR: This research introduces Noise-corrected GRPO (Dr.GRPO), a new framework for Reinforcement Learning from Human Feedback (RLHF) that effectively handles noisy rewards. By modeling reward corruption as Bernoulli noise and applying a novel correction mechanism, the method generates provably unbiased gradient estimates. Experiments on math and code generation tasks demonstrate significant accuracy improvements, showing its ability to mitigate performance degradation caused by inconsistent or erroneous feedback in real-world scenarios.
Reinforcement Learning from Human Feedback (RLHF) and verifiable rewards (RLVR) have become fundamental to developing advanced Large Language Models (LLMs) and state-of-the-art reasoning models. These methods rely heavily on the quality of reward signals, which can often be inconsistent or erroneous, leading to significant performance degradation. This issue, particularly how noise interacts with widely used group-based policy optimization methods, has remained largely unexplored.
A new research paper, “Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradients”, introduces a novel framework designed to tackle this challenge. The authors, Omar El Mansouri, Mohamed El Amine Seddik, and Salem Lahlou, propose a noise-robust Group Relative Policy Optimization (GRPO) and Done Right GRPO (Dr.GRPO) framework that explicitly models reward corruption as Bernoulli noise.
The Challenge of Noisy Rewards
The success of LLM alignment hinges on accurate reward models. However, these models, acting as proxies for human judgment, are susceptible to various forms of noise. This can include inconsistencies in human labeling, approximation errors in the reward model itself, shifts in data distribution, or even adversarial manipulations. When policy optimization algorithms like GRPO or Dr.GRPO are applied directly to these noisy rewards, the learning signal becomes attenuated, leading to suboptimal performance and convergence to a less effective policy.
Introducing Noise-corrected GRPO
The core innovation of this work lies in its ability to debias the learning signal. The framework estimates reward flip probabilities and then applies a correction mechanism, yielding provably unbiased gradient estimates. The theoretical analysis reveals that group-based methods inherently offer some mitigation against individual-level noise, and the proposed correction strategy significantly amplifies this robustness.
Inspired by techniques from supervised learning with noisy labels, the researchers adapt a correction method to the RLHF context. The key idea is to transform the observed noisy reward into a new, debiased random variable whose expectation matches the true, unobserved reward. While simple linear transformations are ineffective in GRPO due to its standardization invariance, the paper introduces a generalized objective and a method to estimate the true variance, allowing for a full recovery of the noiseless GRPO update if noise rates are known.
Practical Implementation and Empirical Success
In real-world scenarios, the true noise rates are unknown and must be estimated. The proposed Algorithm 1 outlines a practical approach: first, estimate the noise rates from a subset of data, and then use these estimates to perform a debiased policy update. For simplicity, the experiments used a single pair of global flip rates across prompts, though future work could explore query-dependent rates.
The empirical validation of Noise-corrected GRPO was conducted in two stages:
1. Synthetic Noise: In a controlled environment where ground truth rewards were available, synthetic noise was introduced. The results showed consistent improvements, with the corrected model sometimes even outperforming the noiseless baseline, suggesting additional regularization benefits.
2. Realistic Estimation: The method was then tested with existing pre-trained reward models on real-world tasks, specifically math reasoning (GSM8K dataset) and code generation (APPS dataset). For math tasks, accuracy improvements of up to 6.7 percentage points were observed. For code generation, gains of up to 1.5 percentage points were achieved, even with a reward model exhibiting extremely high false-positive rates.
These findings highlight the framework’s ability to counteract the negative effects of noisy reward signals, leading to more effective policy optimization. A significant practical advantage is the potential to deploy reinforcement learning in settings with limited annotations, leveraging estimated flip rates to provide scalable, automated feedback without needing extensive curated datasets.
Also Read:
- Bridging Language and Numbers: How New AI Training Boosts LLM Reasoning
- Scaf-GRPO: A Progressive Training Method for Advanced LLM Reasoning
Conclusion and Future Directions
The research demonstrates that noisy rewards significantly weaken the learning signal and lead to suboptimal policy convergence. The Noise-corrected GRPO framework provides a robust solution by offering unbiased gradients and guaranteeing policy improvement. Future work aims to extend this approach to continuous or structured reward signals, handle non-stationary or adversarial noise, improve the efficiency of noise estimation and online training, and apply the framework to multimodal or safety-critical RLHF settings.
