TLDR: A new multi-stage framework developed by researchers from UNSW aims to reduce “hallucinations” (incorrect outputs) in Vision Language Models (VLMs), especially in real-world scenarios like smart glasses. The framework, which secured 3rd place in the KDD Cup 2025 CRAG-MM challenge, prioritizes factual accuracy through lightweight query routing, intelligent retrieval, dual-path answer generation, and a rigorous post-hoc verification process, ensuring more trustworthy AI responses.
In the rapidly evolving world of artificial intelligence, Vision Language Models (VLMs) have shown incredible potential, allowing systems to understand and generate responses from both visual and text inputs. These models are great for tasks like answering questions about images or creating captions. However, a significant challenge they face is “hallucination,” where they confidently produce incorrect or made-up information that isn’t based on the input image or external facts.
This issue is particularly problematic in real-world applications, such as smart glasses, which deal with a constant stream of dynamic, first-person visual data. Users often ask complex, sometimes vague questions that require real-time information and external knowledge for verification. This increases the risk of the VLM making up answers, highlighting the need for systems that are not just capable but also fundamentally reliable.
To tackle this, researchers from The University of New South Wales — Baiyu Chen, Wilson Wongso, Xiaoqian Hu, Yue Tan, and Flora Salim — developed a robust, multi-stage framework. Their solution, detailed in their paper Multi-Stage Verification-Centric Framework for Mitigating Hallucination in Multi-Modal RAG, prioritizes factual accuracy and truthfulness over simply providing a complete answer. This conservative approach is designed to minimize hallucinations, which are heavily penalized in competitive scenarios like the KDD Cup 2025 Meta Comprehensive RAG Benchmark for Multi-modal, Multi-turn (CRAG-MM) challenge.
The Challenge: KDD Cup 2025 Meta CRAG-MM
The CRAG-MM challenge was created to push innovation in Multi-modal Retrieval-Augmented Generation (MM-RAG) systems, especially for scenarios involving egocentric (first-person) and wearable devices. It includes a diverse set of images (egocentric and third-person), over 5,000 question-answer pairs, and mock APIs for image and text-based web searches. The competition had three tasks, progressively evaluating different aspects of MM-RAG systems: single-source augmentation, multi-source augmentation, and multi-turn question answering.
A key aspect of the challenge was its strict evaluation, which heavily penalized incorrect or hallucinatory answers. A single hallucinated answer could negate the benefit of a correct one, strongly encouraging systems to avoid speculation and prioritize truthfulness.
The Solution: A Multi-Stage Verification-Centric Framework
The team’s framework processes each user query through four main stages:
1. Lightweight Query Routing: Not all questions need extensive computation. This initial stage uses a small, efficient model (LLaMA-3.2-1B-Instruct) to quickly classify queries. It determines if external information is needed or if the query is time-sensitive, helping to avoid unnecessary processing and reduce latency, which is crucial for real-time applications like smart glasses.
2. Query-Aware Retrieval and Summarization: Traditional methods often struggle with ambiguous queries and noisy information sources. This framework generates a query-aware image summary using a vision-language model. This summary is then combined with the original question to form a more effective retrieval query. The system uses both text-based and image-based search APIs, followed by a reranking process with a dynamic thresholding method (using Median Absolute Deviation) to filter out irrelevant or noisy information, ensuring only the most relevant snippets are used.
3. Dual-Pathways Generation: To balance relying on retrieved evidence with the model’s existing knowledge, the system generates two responses for each query: one using the retrieved context (RAG) and one without (non-RAG). This allows for a “self-consistency” check, where the system compares the two answers. If they align, it increases confidence in the generated output, indicating it’s both grounded in facts and consistent with the model’s prior knowledge.
4. Post-hoc Verification: Even with dual-path generation, the system could be overly cautious. To address this, a “Chain-of-Verification” (CoV) protocol is used. This involves a two-phase check: a holistic check for overall accuracy and relevance, and a decompositional check where the question is broken into sub-questions, and each sub-answer is independently verified. This process assigns a confidence score to the answer, allowing the system to make a calibrated decision on whether to return the answer or abstain, striking a better balance between reliability and informativeness.
Also Read:
- Connecting Images and Text for Smarter AI: Introducing MMGraphRAG
- AI Agents Collaborate to Understand Long Documents
Results and Impact
The team’s approach achieved 3rd place in Task 1 of the KDD Cup 2025 Meta CRAG-MM challenge. Their ablation study demonstrated the effectiveness of each verification stage in reducing hallucinations and improving factual accuracy. The final system achieved the highest truthfulness score among the evaluated variants, with a significantly lower hallucination rate compared to baselines. This highlights the importance of a verification-centric design in building reliable multi-modal RAG systems for complex, real-world scenarios.
While the team faced limitations, such as hardware constraints preventing the use of multiple VLMs or LoRA fine-tuning, their solution offers valuable insights for developing robust RAG systems in environments like augmented reality/extended reality (AR/XR) and smart assistants.
