TLDR: A new method called Question-Guided Chain-of-Captions (QG-CoC) significantly improves how large multimodal AI models (MLLMs) understand and reason across multiple images. Developed by researchers at UCLA, QG-CoC addresses MLLMs’ limitations in fine-grained perception and multi-image reasoning by breaking down complex questions into sub-questions, generating specific visual captions for each, and then integrating these insights to form a comprehensive answer. This zero-shot prompting approach consistently outperforms existing methods across various benchmarks and models.
Large Multimodal Models (MLLMs) have shown remarkable progress in understanding various forms of data, but they still face significant challenges when dealing with multiple images simultaneously. Researchers Kuei-Chun Kao, Hsu Tzu Yin, Yunqi Hong, Ruochen Wang, and Cho-Jui Hsieh from the University of California, Los Angeles, have introduced a novel approach called Question-Guided Chain-of-Captions (QG-CoC) to address these limitations.
The Core Problem: Multi-Image Understanding
Current MLLMs struggle with two main issues in multi-image contexts: first, a lack of fine-grained perception across different images, and second, a reduced ability to effectively reason over and combine information from multiple visual inputs. While many existing prompting methods focus on describing visual content, most are designed for single-image scenarios or very specific, limited situations. This leaves a gap in how MLLMs handle more general and complex tasks that involve reasoning across several images.
The research began with an extensive investigation into how current prompting methods perceive detailed visual information and process it when multiple images are involved. The findings revealed that existing methods often fail to focus on necessary clues and seamlessly integrate perception with reasoning.
Insights from Preliminary Studies
To develop QG-CoC, the researchers first analyzed different captioning strategies. They found three key insights:
- Detailed Captions are Better: Providing detailed descriptions for images, rather than concise ones, significantly improves multi-image understanding. Detailed captions help prevent information loss, which is crucial for complex reasoning.
- Individual Captions for Each Image: When dealing with multiple images, generating a separate caption for each image is more effective than creating a single summarized caption for the entire set. This ensures precise, image-specific information is captured.
- Question-Guided Captions: Integrating the main question into the caption generation process leads to better performance. This ensures that the generated captions are directly relevant to the task at hand, focusing on important visual elements.
Furthermore, the study examined how existing single-image prompting methods, such as DDCoT, CCoT, and CoCoT, perform when adapted to multi-image scenarios. It was observed that while these methods could identify individual entities and straightforward relationships, they struggled with extracting deeper, implicit context or performing complex reasoning that requires synthesizing information from an arbitrary number of images. For instance, they often lacked spatial context, provided unrelated object descriptions, or gave vague comparisons.
Introducing QG-CoC: A Three-Step Approach
Inspired by these findings, QG-CoC was developed as a zero-shot prompting method that can handle any number of images. It follows a structured, three-step reasoning process:
1. Decompose the Question: The first step involves breaking down a complex main question into a series of simpler, more manageable sub-questions. Each sub-question targets a specific aspect of the image(s), ensuring that the reasoning is detailed and aligned with the original question’s intent.
2. Caption Key Information for Each Sub-Question: For each sub-question, the MLLM generates targeted captions. These captions extract and describe the most relevant visual evidence, such as objects, actions, or scene changes, providing intermediate interpretations that directly link reasoning to the image content.
3. Answer Sub-Questions and Integrate Reasoning: Finally, the model answers each sub-question based on its specific captions, forming a coherent chain of reasoning. These individual answers are then combined to produce the final response to the original question, supported by clear visual evidence.
Experimental Validation and Performance
The researchers evaluated QG-CoC on various open-source (LLaVA-OneVision, Mantis-idefics2, Qwen-2.5-VL) and closed-source (GPT-4o, Gemini-1.5-Flash) MLLMs using multi-image benchmarks like MuirBench and MMIU, as well as single-image tasks. The results consistently showed that QG-CoC outperforms existing prompting methods in multi-image scenarios and also demonstrates strong generalization in single-image tasks.
Compared to methods that provide general detailed captions, QG-CoC’s targeted, sub-question-guided captions lead to more focused and effective reasoning. While other methods like DDCoT also decompose questions, QG-CoC’s crucial intermediate step of generating explicit captions for each sub-question provides a stronger grounding in visual evidence, resulting in more robust reasoning.
Also Read:
- Advancing Vision-Language Models with Multi-Prompt Learning
- Unpacking How AI Models Resolve Conflicting Information from Images and Text
Understanding Limitations and Future Directions
An analysis of errors revealed that the most common issues were wrong reasoning (35%), followed by inaccurate question understanding (33.3%), and inaccurate perception (31.7%). This suggests that even with accurate context, MLLMs can still struggle with inferring the correct reasoning path. Tasks requiring holistic multi-image understanding were particularly prone to reasoning errors, while fine-grained perception tasks faced more perception errors.
While QG-CoC introduces some computational overhead due to its two-stage pipeline, the researchers argue that this is a justifiable trade-off for the consistent and significant performance improvements it delivers, especially for open-source models.
In conclusion, QG-CoC offers a powerful new baseline for multi-image reasoning in MLLMs by effectively balancing detail, specificity, and relevance through its question decomposition and guided captioning approach. The full research paper can be accessed here: QG-CoC: Question-Guided Chain-of-Captions for Large Multimodal Models.
