TLDR: MI-RAG is a novel framework that significantly improves how AI models answer complex visual questions requiring external knowledge. Unlike traditional single-pass methods, MI-RAG iteratively refines its reasoning and dynamically searches for information across both visual and textual knowledge bases using a ‘reasoning-guided multi-query’. This approach leads to enhanced retrieval recall and answer accuracy on challenging benchmarks, demonstrating a scalable solution for compositional reasoning in knowledge-intensive Visual Question Answering.
Multimodal Large Language Models (MLLMs) have made significant strides in understanding and processing information from both images and text. However, they often face challenges when answering complex visual questions that require external knowledge beyond what’s immediately visible in an image. This is where Retrieval-Augmented Generation (RAG) comes in, a promising solution that provides models with access to external knowledge bases.
Traditional RAG systems typically operate in a single-pass, retrieve-then-read fashion. This means they make one attempt to gather relevant information and then synthesize an answer. While effective to some extent, this approach can fall short when questions are particularly knowledge-intensive, as a single retrieval might not capture all the necessary facts, and a single reasoning step can be misled by irrelevant information.
To address these limitations, researchers Changin Choi, Wonseok Lee, Jungmin Ko, and Wonjong Rhee have introduced a novel framework called MI-RAG: Multimodal Iterative RAG. This innovative system moves beyond the single-pass approach by employing an iterative process that continuously refines its reasoning and retrieval across multiple rounds. The core idea is to leverage the model’s evolving understanding to dynamically formulate better queries and integrate newly acquired knowledge.
How MI-RAG Works
MI-RAG operates in a cyclical manner. It starts by generating an initial understanding or ‘reasoning record’ based on the input image and question. In each subsequent iteration, this accumulated reasoning record is used to create a ‘multi-query.’ This multi-query isn’t just one question; it’s a set of complementary queries designed to explore different facets of the visual entity and related textual knowledge.
These dynamic queries then drive a joint search across two types of knowledge bases: a multimodal knowledge base (containing image-text pairs for visual grounding) and a textual knowledge base (offering broader textual information). This dual-source approach ensures a diverse and comprehensive collection of factual links. The newly retrieved knowledge is then synthesized and incorporated into the reasoning record, progressively enhancing the model’s understanding. This iterative refinement continues for a set number of cycles, leading to a more robust and accurate final answer.
Also Read:
- AnchorRAG: A Multi-Agent Framework for Enhanced Open-World Question Answering with Knowledge Graphs
- Boosting 3D CAD Part Retrieval with Reflective AI: A Training-Free Framework
Key Contributions and Benefits
The MI-RAG framework makes two primary contributions. First, its reasoning-guided multi-query dynamically searches for information across different modalities, ensuring a comprehensive knowledge gathering process. Second, the joint search across heterogeneous knowledge bases allows the model to compose visually-grounded knowledge with broader textual context, which is crucial for complex questions.
The researchers conducted experiments on challenging benchmarks like Encyclopedic VQA, InfoSeek, and OK-VQA. The results demonstrated that MI-RAG significantly improves both the recall of retrieved information and the accuracy of the answers. It also showed strong scalability, meaning its performance further improves with more capable underlying MLLMs (like Gemini-2.5-Flash) and more powerful retrievers.
Ablation studies, where components of the system are removed to see their impact, confirmed the importance of each part of MI-RAG. The reasoning-guided multi-query, the use of heterogeneous knowledge bases, and especially the iterative process itself, all contribute significantly to the framework’s superior performance. The iterative refinement, while incurring some computational cost, consistently improves accuracy and recall over successive steps.
In conclusion, MI-RAG presents a scalable and effective paradigm for advancing compositional reasoning in knowledge-intensive Visual Question Answering, offering a promising direction for future research in multimodal AI. You can read the full paper here.
