TLDR: A new method called Off-Policy Corrected Reward Modeling (OCRM) improves how large language models (LLMs) learn from human feedback. It tackles “overoptimization,” where the model gets better at maximizing a reward but actually gets worse at matching human preferences. OCRM fixes this by periodically updating the reward model using a technique called importance weighting, which re-calibrates it to the model’s current behavior without needing new human data. This leads to more accurate reward models and better-performing LLMs in tasks like summarization and chatbots.
Reinforcement Learning from Human Feedback (RLHF) has become a crucial technique for training large language models (LLMs) to align with complex human preferences. This process typically involves three main steps: supervised fine-tuning (SFT) of a language model, collecting human feedback on pairs of responses to train a reward model (RM), and finally, using reinforcement learning (RL) to train the LLM to maximize the reward given by this RM.
However, a significant challenge arises during the RL phase, known as ‘overoptimization’ or ‘Goodharting’. As the LLM continues to train and generate responses that increasingly differ from the initial responses the RM was trained on, the RM can become inaccurate. This leads to a situation where the reward score given by the RM keeps increasing, but the actual quality of the responses, as judged by humans, stagnates or even declines. This issue is fundamentally a ‘distribution shift’ problem, where the data distribution the RM was trained on no longer matches the distribution of responses generated by the evolving LLM.
Researchers have investigated this overoptimization phenomenon from the perspective of distribution shift. They found that this shift results in an inconsistent estimate of the RM’s parameters, which in turn leads to an inconsistent estimate of the policy gradient—the direction in which the LLM is updated. This means that standard RLHF methods may not converge to the truly optimal policy, even with unlimited data and training.
Introducing Off-Policy Corrected Reward Modeling (OCRM)
To address this critical issue, a new method called Off-Policy Corrected Reward Modeling (OCRM) has been proposed. OCRM iteratively corrects the reward model using a technique called importance weighting (IW). The beauty of this approach is that it doesn’t require collecting new human labels or samples, which are typically very costly and time-consuming.
The core idea behind OCRM is to re-weight the original dataset used to train the RM. By knowing the probability ratio between the current policy’s outputs and the initial SFT policy’s outputs, OCRM can effectively make the original data look like it came from the current, evolving policy. This allows the RM to be retrained to be accurate for the current policy’s outputs.
Since the LLM’s policy changes with each update, ideally, the RM would need to be retrained after every single policy update. While this is computationally infeasible, OCRM implements an approximate method: it retrains the RM using importance weighting after a set number of policy updates (denoted as ‘k’ updates). Additionally, OCRM also updates the reference for the KL-regularization term, which is typically used to keep the LLM close to its initial SFT distribution. Instead, it now keeps the LLM close to the *previous* policy’s distribution, ensuring the model remains in a region where the RM is accurate.
Also Read:
- Personalizing AI: How Text Summaries Help Language Models Understand You Better
- AI’s Achilles’ Heel: Why More Feedback Can Harm Large Language Model Performance
Empirical Validation and Performance
The effectiveness of OCRM was validated through experiments on two common language model alignment tasks: TL;DR summarization and a chatbot task using a length-truncated version of the Alpaca-Farm dataset. The results showed that OCRM significantly outperforms standard RLHF methods such as PPO-RLHF, Direct Preference Optimization (DPO), Weighted Preference Optimization (WPO), and Reward Learning on Policy (RLP-SPG).
For instance, in summarization tasks, OCRM achieved higher win rates against reference responses as judged by a powerful ‘gold RM’ (a proxy for human feedback). Ablation studies further demonstrated that both the off-policy correction and the dynamic updating of the KL-regularization reference contribute to the improved performance. The method also proved robust even with smaller training datasets and showed consistent improvements when evaluated with feedback from GPT 4.1 Nano, a more realistic synthetic setup.
While OCRM introduces additional computational cost due to RM retraining, this cost is relatively small compared to the main RL training steps, which involve generating new completions autoregressively. The method’s ability to achieve better alignment without requiring new human feedback makes it a promising advancement in the field of LLM alignment.
For more technical details, you can refer to the full research paper: Off-Policy Corrected Reward Modeling for Reinforcement Learning from Human Feedback.
