TLDR: Cooper is a new reinforcement learning framework for large language models that simultaneously trains both the language model (policy) and its reward system. This approach tackles common issues like “reward hacking” (where the model learns to trick a fixed reward system) and improves overall performance in reasoning tasks by dynamically adapting the reward model alongside the language model.
Large language models (LLMs) have shown incredible abilities in complex reasoning tasks, from mathematics to coding. A key technique for boosting these capabilities is reinforcement learning (RL), where models learn by receiving feedback, or “rewards,” for their actions. However, current reward systems in RL for LLMs face significant challenges.
The Problem with Current Reward Systems
Traditionally, RL for LLMs uses two main types of reward systems: model-based and rule-based. Model-based rewards, which are calculated dynamically by another model, are prone to a problem called “reward hacking.” This is where the language model learns to exploit weaknesses in the fixed reward system, getting high scores even if its answers aren’t truly correct. Imagine a student who learns to game the grading system rather than actually understanding the material. This can lead to catastrophic failures in training.
On the other hand, rule-based rewards rely on predefined rules to check answers. While less susceptible to hacking, they often lack robustness. They might struggle with diverse answer formats or subtle variations, leading to incorrect judgments and limiting how much the language model can improve.
Introducing Cooper: A Co-Optimization Framework
To address these limitations, researchers have introduced Cooper (Co-optimizing Policy Model and Reward Model), a novel RL framework that jointly optimizes both the language model (referred to as the policy model) and the reward model. Cooper’s design aims to combine the best of both worlds: the high precision of rule-based rewards for identifying truly correct answers and the flexibility of model-based rewards.
Cooper works by continuously refining the reward model during the RL process. It does this by dynamically creating and selecting pairs of correct and incorrect examples. For instance, it uses highly precise rule-based checks to identify genuinely correct responses as “positive” examples. For “negative” examples, it employs an assistant LLM to intentionally transform correct answers into plausible but incorrect ones, ensuring the reward model learns to spot subtle errors.
VerifyRM: A Smarter Reward Model
A crucial component supporting Cooper is VerifyRM, a new reference-based reward model. Unlike typical reward models that only look at the question and the model’s answer, VerifyRM also takes a “reference answer” as input. This additional context significantly improves its accuracy in verifying answers for reasoning tasks. VerifyRM was trained on a massive dataset of mathematical problems and solutions generated by various LLMs, using a clever “hybrid annotation” strategy that combines rule-based verifiers with LLMs acting as judges, all without needing extensive manual labeling.
How Cooper Prevents Reward Hacking
Experiments with Cooper have shown remarkable results. While language models trained with static reward models suffered a significant performance drop (up to 16% in some cases) due to reward hacking, Cooper not only prevented this collapse but also achieved superior performance. This is because as the language model learns and evolves, the reward model in Cooper adapts its understanding of what constitutes a good answer, effectively closing off opportunities for the language model to exploit a fixed system.
The research highlights that treating reward models as dynamic, evolving components is crucial for stable and effective reinforcement learning with LLMs. This co-optimization approach suggests that many perceived instabilities in RL might stem from reward exploitation rather than fundamental optimization challenges.
Also Read:
- Enhancing LLM Reasoning with Consistency-Aware Policy Optimization
- Language Models Learn Through Self-Generated Questions
Future Directions
While Cooper represents a significant step forward, the researchers acknowledge areas for future improvement. These include reducing its dependency on specific domain verification tools, addressing potential computational overhead from dual optimization, and exploring ways to generate negative samples without relying on an external assistant LLM. Nevertheless, Cooper establishes a promising direction for developing more robust and accurate RL training paradigms for large language models. For more details, you can read the full research paper here.
