TLDR: AdaPO is a new reinforcement learning framework for Large Multimodal Models (LMMs) that helps them self-evaluate and improve their outputs. It uses an Adaptive Reward Model and a dynamic regularization mechanism to prevent issues like “reward hacking” and training instability. This leads to better accuracy and self-correction capabilities in a single, automated training process, outperforming previous methods across various benchmarks.
Large Multimodal Models (LMMs) are advanced AI systems capable of understanding and processing information from various sources, like text and images. They have shown impressive abilities in complex tasks, especially when using Chain-of-Thought (CoT) reasoning, which involves breaking down problems into steps. However, a significant challenge for these models is their ability to self-evaluate—to assess the correctness of their own outputs and improve them in multi-turn conversations.
Current methods often struggle with this self-evaluation. Many foundation models lack this crucial capability. While reinforcement learning (RL) has been used to enhance self-evaluation, it often faces a problem called “reward hacking.” This occurs when a fixed reward system encourages the model to optimize for a single objective, leading to unintended behaviors or a decline in overall performance, sometimes even causing model collapse.
To address these issues, researchers have proposed a new online reinforcement learning framework called AdaPO, which stands for Adaptive Policy Optimization. AdaPO is designed to dynamically adjust its training objectives in real-time based on the model’s current training state for each task. This adaptive approach helps mitigate the problem of reward hacking and enhances training stability.
How AdaPO Works
AdaPO introduces two main innovations:
- Adaptive Reward Model (ARM): This model dynamically assesses the task’s training state by looking at the performance distribution of the model’s generated multi-turn trajectories. It adjusts the reward allocation across different types of trajectories—for example, whether an answer was initially correct, successfully corrected, or initially incorrect and failed to be corrected. By balancing positive and negative rewards, ARM prevents the model from over-optimizing for a single objective, which is a common cause of reward hacking.
- Reward-Aware Dynamic KL Regularization: This mechanism replaces a fixed penalty with dynamic coefficients. These coefficients are modulated by the “reward gap” between different multi-turn situations. For instance, if the reward for correcting an error is significantly higher, it imposes a stronger constraint on the erroneous reasoning path, helping to stabilize the model’s direct CoT generation while still allowing for effective learning.
A notable feature of AdaPO is its ability to automatically and smoothly adjust its learning focus based on the training progress of sub-tasks, without requiring any manual intervention. This makes the training process more efficient and stable.
Also Read:
- Boosting Recommendation Accuracy with Smarter Negative Sample Handling in LLMs
- Boosting LLM Reasoning: A New Approach to Overcome Learning Plateaus
Experimental Results and Impact
Extensive experiments were conducted across eight different benchmarks, using various models. The results showed that AdaPO significantly improves both the direct reasoning and self-evaluation capabilities of Large Multimodal Models. For instance, it achieved a relative improvement in self-evaluation capability of up to 25.5% and outperformed all baselines on 93.75% of tasks.
Compared to other training methods like SFT (Supervised Fine-Tuning) and staged RL training methods (like GRPO and SCoRe), AdaPO demonstrated superior performance. SFT-based methods often failed to learn effective self-evaluation, showing high misjudgment rates. Staged RL methods could improve self-evaluation but often at the cost of reduced direct response accuracy. AdaPO, however, managed to simultaneously enhance both the model’s direct response accuracy and its self-evaluation capability, achieving the lowest misjudgment rate and significant accuracy improvements.
The generality of AdaPO was also confirmed across different model scales, including Qwen2-VL-2B and Qwen2-VL-7B, consistently achieving higher accuracy for both initial and self-evaluated responses.
Ablation studies further validated the contribution of each component within AdaPO. Removing the Adaptive Reward Model led to a significant decrease in self-correction capability. Without Reward-Aware KL, the model’s initial accuracy decreased, and its ability to improve through self-evaluation was reduced. Additionally, data filtering strategies were found to be crucial for improving training efficiency and overall model performance by eliminating redundant or harmful training data.
In conclusion, AdaPO offers a robust solution to the challenges of self-evaluation in LMMs. By dynamically adjusting its training objective, it effectively mitigates reward hacking and training instability, allowing models to maintain a crucial balance between correcting errors and preserving correct answers. This approach represents a promising direction for future research in reinforcement learning for self-improving AI systems. For more technical details, you can refer to the full research paper: A Rolling Stone Gathers No Moss: Adaptive Policy Optimization for Stable Self-Evaluation in Large Multimodal Models.
