TLDR: The DB3 team secured the grand prize at the Meta CRAG-MM Challenge 2025 by developing a comprehensive multi-modal, multi-turn question answering system. Their solution integrated domain-specific retrieval pipelines for image-indexed knowledge graphs, web sources, and conversational contexts with a unified LLM-tuning approach for hallucination control, including advanced refusal training. They achieved top rankings across all three tasks, demonstrating particular strength in handling ego-centric queries.
The Meta CRAG-MM Challenge 2025, hosted at KDD Cup’25, aimed to push the boundaries of multi-modal reasoning in artificial intelligence. This competition introduced a novel benchmark, CRAG-MM, designed to evaluate factual question answering in complex multi-modal contexts, integrating images with multi-turn conversational flows. A key characteristic of this challenge was the focus on ego-centric images—pictures taken from a first-person perspective, often using wearable cameras. These images, while rich in contextual information, present unique challenges like frequent occlusions and rapid viewpoint shifts.
The challenge was structured into three distinct tasks:
Task 1: Single-turn questions with image-indexed knowledge graphs.
Task 2: Single-turn questions with an additional web search knowledge source.
Also Read:
- HANRAG: Enhancing AI’s Question Answering Through Adaptive Retrieval and Noise Filtering
- AI Unites Visuals and Policies for Smarter Disaster Damage Assessment
Task 3: Multi-turn retrieval-augmented generation (RAG) tasks involving dialogue-like interactions and information integration from multiple sources.
The DB3 team emerged as a leading contender, securing 2nd place in Task 1, 2nd place in Task 2, and an impressive 1st place in Task 3. This outstanding performance led them to win the grand prize for excellence in handling ego-centric queries, demonstrating superior capabilities in first-person perspective challenges.
The team’s winning strategy, detailed in their paper DB3 Team’s Solution For Meta KDD Cup’ 25, centered on a comprehensive framework. This framework integrated specialized retrieval pipelines tailored for different tasks with a unified Large Language Model (LLM) tuning approach specifically designed for hallucination control. The solution’s core features included domain-specific retrieval pipelines capable of handling image-indexed knowledge graphs, web sources, and multi-turn conversations. Additionally, it incorporated advanced refusal training using Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Reinforcement Learning (RL) to minimize incorrect answers.
A significant challenge addressed by the DB3 team was the effective retrieval of information from diverse sources. For image-indexed knowledge graphs (Task 1), they tackled the difficulty of ego-centric images, which often contain irrelevant backgrounds compared to Wikipedia-style index images. They introduced ‘grounding in the query image’ using the Grounding-DINO model to localize the correct entity within the query image. They also explored reranking through image comparison and text-based retrieval methods, such as extracting entity names and using merged text queries to improve accuracy.
For text-indexed web sources (Task 2), the team focused on rewriting queries into a ‘merged text query’ that combined visual information from the image with the original text query. This allowed for more effective retrieval from text-based web content. They utilized SFT tuning for this rewrite process and experimented with RL tuning approaches like DPO and GRPO, although SFT-tuning proved most effective for their final submission.
In the multi-turn conversation task (Task 3), the team adapted their merged query rewrite process to incorporate conversational context, typically using the last-step context for simplicity and efficiency. They also considered integrating OCR (Optical Character Recognition) for text-heavy images but ultimately decided against it due to the computational cost and limitations of available open-source tools for ego-centric images.
A notable component of their solution was the use of specialized tools for solving mathematical problems. Recognizing the inherent difficulties LLMs and VLMs face with math, they developed tools for calculations, base conversions, and chemical formula balancing, training their VLM to effectively use these APIs.
Hallucination control was another critical aspect, given the contest’s scoring system that penalized incorrect answers. The team aimed to maximize correct answers while minimizing wrong ones by training the model to output “I don’t know” when uncertain. Their refusal training pipeline involved SFT, DPO, and GRPO, each contributing to the model’s ability to estimate answerability and refuse confidently.
Finally, the DB3 team employed sophisticated checkpoint ensemble tricks to combine and select the best models. They explored strategies like ensembling according to domain or equivalence, and mixed approaches. While these ensemble methods significantly boosted local performance, they also highlighted challenges with overfitting and time limits during online evaluation, leading to a simpler final version.
In conclusion, the DB3 team’s success in the Meta CRAG-MM Challenge 2025 underscores the importance of combining task-specific retrieval mechanisms with robust hallucination control for developing reliable multi-modal question-answering systems, especially in challenging scenarios involving ego-centric visual data.
