TLDR: Omni-Reward introduces a unified framework for omni-modal reward modeling, addressing the limitations of current reward models in handling diverse data modalities (text, image, video, audio, 3D) and rigid human preferences. It comprises Omni-RewardBench, a comprehensive evaluation benchmark; Omni-RewardData, a large-scale multimodal preference dataset with instruction-tuning; and Omni-RewardModel, a family of discriminative and generative reward models. The framework demonstrates strong performance and generalization, highlighting the importance of mixed multimodal data and instruction-tuning for aligning AI with complex human feedback.
Artificial intelligence is rapidly advancing towards a future where models can process and generate information across all types of data, known as the omni-modal paradigm. This means AI should be able to understand and create content using text, images, video, audio, and even 3D data. A crucial part of making AI behave in ways humans prefer is through ‘reward models’ (RMs), which learn from human feedback to guide AI behavior.
However, current reward models face two significant hurdles. First, there’s a ‘Modality Imbalance,’ meaning most RMs are primarily designed for text and images, with limited support for other important modalities like video, audio, and 3D. Second, they suffer from ‘Preference Rigidity,’ as they are often trained on simple binary choices (like ‘good’ or ‘bad’), which fails to capture the rich and diverse nature of human preferences.
Introducing Omni-Reward: A Comprehensive Solution
To tackle these challenges, researchers have introduced Omni-Reward, a new framework designed for generalist omni-modal reward modeling that supports free-form preferences. This framework consists of three main components:
1. Omni-RewardBench (Evaluation): This is the first benchmark specifically created to evaluate omni-modal reward models using free-form preferences. It covers nine different tasks across five modalities: text, image, video, audio, and 3D. This benchmark helps assess how well RMs perform when dealing with diverse data types and complex human instructions. For example, it includes tasks like generating text from text, images from text, or even 3D objects from text, and then evaluating how well the AI’s output aligns with specific human criteria.
2. Omni-RewardData (Data): To train these advanced reward models, a new large-scale multimodal preference dataset called Omni-RewardData has been constructed. It includes 248,000 general preference pairs and an additional 69,000 instruction-tuning pairs. The instruction-tuning data is particularly important because it helps RMs understand and adapt to user preferences expressed in natural, free-form language, moving beyond rigid binary choices.
3. Omni-RewardModel (Model): The framework also proposes a new family of reward models. This includes a ‘discriminative’ RM (Omni-RewardModel-BT) that assigns a scalar score to responses, and a ‘generative’ RM (Omni-RewardModel-R1) that not only provides a score but also generates a textual explanation or ‘critic’ for its decision, making the scoring process more transparent and understandable. These models are trained using the extensive Omni-RewardData.
Key Findings and Performance
Experiments conducted using Omni-RewardBench revealed several important insights. Current multimodal large language models (MLLMs) and specialized RMs still face significant challenges, especially when dealing with the more complex ‘with ties’ evaluation setting. The ‘modality imbalance’ problem was clearly evident, with existing models performing poorly on tasks involving audio generation (text-to-audio), 3D generation (text-to-3D), and complex image editing (text-image-to-image).
However, the proposed Omni-RewardModel showed strong performance on Omni-RewardBench, achieving high accuracy and demonstrating good generalization across challenging tasks. It also performed comparably to, or even better than, state-of-the-art models on other widely used reward modeling benchmarks like VL-RewardBench and Multimodal RewardBench. A crucial finding was that instruction-tuning data is vital for RMs, as it helps them overcome ‘preference rigidity’ and dynamically adjust reward scores based on specific user preferences.
Further analysis showed that training with mixed multimodal data significantly improves generalization across tasks. There’s also a notable correlation in performance among similar tasks; for instance, understanding tasks (text, image, video understanding) show strong correlations, as do generation tasks (video, 3D, image generation).
The research also explored the impact of ‘Chain-of-Thought’ (CoT) reasoning, where models explain their decisions. CoT was found to boost the performance of weaker models by providing intermediate reasoning steps, though it had less impact on already strong models. Additionally, a ‘pairwise’ scoring strategy (directly comparing two responses) proved more effective than a ‘pointwise’ approach (scoring each response individually).
Also Read:
- Self-Rewarding PPO: Improving LLM Generalization from Demonstrations
- RobotArena∞: A New Framework for Scalable Robot Evaluation
The Path Forward for AI Alignment
In conclusion, Omni-Reward represents a significant step towards creating more generalist and human-aligned AI models. By addressing modality imbalance and preference rigidity through its innovative benchmark, dataset, and models, it paves the way for AI systems that can better understand and respond to diverse human preferences across all forms of media. For more technical details, you can refer to the full research paper here.
