TLDR: MAP (Map-Level Attention Processing) is a novel, training-free method designed to mitigate hallucinations in Large Vision-Language Models (LVLMs). It reinterprets the model’s hidden states as a 2D semantic map, recognizing that factual information is broadly distributed. By employing Layer-Wise Criss-Cross Attention to refine token representations and Global-Local Logit Fusion to combine predictions, MAP significantly improves factual consistency and performance across various benchmarks without requiring additional model training.
Large Vision-Language Models, or LVLMs, have made incredible strides in understanding and generating content based on both images and text. They can answer questions about images, describe scenes, and perform many other complex tasks. However, these powerful AI models still face a significant challenge: hallucinations. This means they sometimes generate information that sounds grammatically correct but is factually inaccurate or inconsistent with the visual input, like describing an object that isn’t actually in the picture.
Addressing these hallucinations is crucial, especially for applications where accuracy is paramount, such as medical imaging or autonomous driving. While many existing methods try to reduce hallucinations, they often involve computationally expensive fine-tuning or focus on limited aspects of the model’s internal workings. These methods typically look at information either across different layers of the model (inter-layer) or within a single layer (intra-layer).
A New Perspective: The Semantic Map
Researchers have introduced a novel approach called Map-Level Attention Processing, or MAP, which offers a fresh perspective on tackling hallucinations. Instead of focusing on isolated layers or regions, MAP interprets the entire set of a model’s hidden states—the internal representations it builds as it processes information—as a two-dimensional semantic map. Think of it like a detailed landscape where every ‘pixel’ is a hidden state token. The key insight here is that factual information isn’t confined to small, localized areas; it’s widely distributed across this entire semantic map.
This observation is backed by analysis showing that even intermediate hidden states retain factual signals. For instance, if an image contains a ‘bed’, certain tokens across various layers and positions within the model’s internal map will show high probabilities for the word ‘bed’, while non-existent objects consistently receive low confidence scores. This suggests a rich, distributed source of factual cues that can be leveraged.
How MAP Works
MAP is a training-free decoding method, meaning it doesn’t require additional training of the model, making it more efficient for real-world use. It incorporates two main components:
-
Layer-Wise Criss-Cross Attention: This mechanism refines the token representations at each decoding layer. It does so by aggregating information from both inter-layer (across different layers) and intra-layer (within the same layer) dimensions. This allows the model to gather factual signals from a broader ‘neighborhood’ within the semantic map, leading to more accurate and consistent representations.
-
Global-Local Logit Fusion: After refining the hidden states, MAP further enhances the final prediction. It combines the predictions (logits) obtained from both a ‘local’ perspective (the refined token itself) and a ‘global’ perspective (information aggregated from the entire semantic map). This fusion balances fine-grained local evidence with broader contextual information, making the model’s outputs more robust and accurate.
Also Read:
- Improving AI Explanations: CoRGI Introduces Visual Grounding to Chain-of-Thought
- Visual Prompts: A Double-Edged Sword for AI Vision Models
Demonstrated Effectiveness
Extensive experiments have shown that MAP significantly reduces visual hallucinations and improves factual accuracy across various LVLMs, including popular models like LLaVA1.5, mPLUG-Owl2, and InstructBLIP. It has been evaluated on several benchmarks specifically designed to test for hallucinations, such as POPE, MME, and MMHal-Bench.
On the MME benchmark, MAP consistently achieved the highest total scores, demonstrating notable performance gains compared to regular decoding. Similarly, on MMHal-Bench, which assesses both informativeness and hallucination severity, MAP showed stable improvements across most tasks, outperforming other methods. For object-level hallucination evaluation on POPE, MAP also consistently outperformed most baselines, especially on challenging adversarial subsets.
These results highlight the robustness and effectiveness of the map-level decoding strategy in mitigating hallucinations. By treating the model’s internal states as a comprehensive 2D semantic map and leveraging information across its full extent, MAP offers a promising direction for making LVLMs more truthful and reliable.
For more technical details, you can refer to the full research paper here.
