TLDR: VERITAS is a new pipeline that significantly improves the quality of training data for large multimodal AI models (LMMs). It achieves this by integrating structured visual information (vision priors), critiques from multiple state-of-the-art LMMs (like GPT-4o, Gemini-2.5-Pro), and statistical fusion to create highly reliable ‘gold standard’ answers. A lightweight, efficient AI critic model is then trained to evaluate and select the best refined answers. This process leads to substantial performance gains for LMMs, especially in visual perception and reasoning tasks, while also reducing factual errors and hallucinations.
The performance of today’s advanced artificial intelligence models, particularly Large Multimodal Models (LMMs) that understand both images and text, heavily relies on the quality of their training data. However, existing methods for creating this data often fall short, leading to models that make factual errors or ‘hallucinate’ information due to inadequate visual understanding.
Addressing this critical challenge, researchers have introduced VERITAS, a novel pipeline designed to systematically enhance the quality of supervised fine-tuning (SFT) data for LMMs. VERITAS stands for Vision-Priors Evaluation and Refinement through Integration of Tri-Expert Assessment with Shrinkage, and it promises to make multimodal AI models more accurate and reliable.
How VERITAS Works: A Four-Stage Process
VERITAS operates through four interconnected stages, each contributing to a more robust and accurate dataset:
1. Vision-Prior Extraction: The process begins by grounding each image with explicit perceptual evidence. Specialized visual recognition models like RAM++ and OCR systems such as PP-OCRv4 are used to extract structured information, including object tags and text strings (from OCR). This visual data acts as a ‘prior,’ providing concrete evidence that subsequent stages can reference, ensuring critiques are based on observable facts.
2. Tri-Expert Assessment with Shrinkage Fusion: In this stage, three leading LMMs—GPT-4o, Gemini-2.5-Pro, and Doubao-1.5-pro—independently evaluate the quality of answers associated with the images and questions. Each expert provides a critique and a numerical score. To overcome individual biases and reduce variance, these scores are then statistically fused using a sophisticated ‘domain-aware James-Stein–style shrinkage’ method. This process generates a highly confident ‘gold score’ that serves as a reliable ground truth for answer quality.
3. Integration with GRPO Critic: The valuable insights from the costly multi-expert assessment are then distilled into a lightweight 7-billion-parameter critic model. This critic is trained using Group Relative Policy Optimization (GRPO), a method that enables it to efficiently evaluate answers based on reasoning. The GRPO critic achieves a ranking fidelity comparable to state-of-the-art LMMs like GPT-4o but at a significantly lower computational cost, making the evaluation process much more scalable.
4. Self-Refinement and Answer Selection: Finally, based on the expert critiques and the high-confidence gold score, the expert LMMs generate revised candidate answers. A pool of answers, including the original and the newly generated ones, is then presented to the GRPO critic. The critic rescores each candidate, and the highest-scoring answer is selected as the final, refined answer for the dataset. This approach ensures that data quality is enhanced without discarding valuable samples, maintaining the diversity and richness of the training data.
Also Read:
- AI Agents Learn and Adapt Through Dialogue to Tackle Complex Problems
- Optimizing AI Reasoning for Shorter, Smarter Responses
Impact and Benefits
Extensive experiments across six multimodal benchmarks have demonstrated the effectiveness of VERITAS. Models fine-tuned with data processed by VERITAS consistently outperform those trained on raw data, showing an average accuracy improvement of +7.4%. These gains are particularly significant in tasks requiring precise visual perception, such as OCR-VQA (text-in-image understanding) and MME (basic perception diagnostics), where improvements of +14.35 and +14.2 points were observed, respectively.
The vision priors play a crucial role in these improvements, providing explicit low-level evidence that helps the critic identify subtle errors. Furthermore, the multi-expert scoring and diversity-driven rewriting lead to better cross-modal understanding and reasoning capabilities. VERITAS also shows a strong ability to mitigate hallucinations, ensuring that refined answers do not introduce new unsupported statements.
A key achievement is the lightweight GRPO critic, which not only rivals the ranking fidelity of GPT-4o but does so at a fraction of the computational cost, making high-quality data curation more accessible. The domain-aware fusion method also proves superior to simple averaging, providing a more calibrated and reliable training signal.
The researchers have made the VERITAS pipeline, a 96K confidence-annotated multimodal dataset, and all model checkpoints publicly available to foster further research in multimodal data optimization. This work represents a significant step towards building more reliable and accurate large multimodal models. You can read the full research paper here.
