TLDR: FOCUS is a new training-free decoding strategy that improves Large Vision-Language Models’ (LVLMs) ability to handle multiple images by preventing “cross-image information leakage.” It works by sequentially masking all but one image with noise, performing focused inference, and then contrastively aggregating the results to ensure the model accurately understands each image independently without mixing visual cues from others. This method consistently boosts performance on multi-image tasks across various LVLMs without requiring retraining or architectural changes.
Large Vision-Language Models (LVLMs) have shown impressive capabilities in understanding both visual and textual information, excelling in tasks like Visual Question Answering and Image Captioning. However, their performance often falters when dealing with multiple images simultaneously. Researchers have identified a key issue: “cross-image information leakage,” where visual cues from different images become mixed in the model’s output.
Imagine showing an LVLM two images: one with “beer and bananas” and another with “banana and coffee container.” When asked about the first image, a typical LVLM might incorrectly describe it as containing “beer, banana, and coffee container,” pulling in elements from the second image. This happens because current LVLMs, despite being given multiple inputs, struggle to process each image independently. They tend to entangle information, leading to inaccurate descriptions that combine content from various sources.
Addressing this challenge is crucial for advancing LVLMs into more complex, real-world applications that involve multiple visual inputs. While some existing methods attempt to improve multi-image reasoning through extensive training on large datasets, these approaches are often resource-intensive and lack flexibility. Other inference-time strategies exist, but they are typically designed for single-image understanding and require significant architectural modifications, making them difficult to integrate broadly.
To overcome these limitations, a new decoding strategy called FOCUS has been proposed. FOCUS is a training-free and architecture-agnostic solution designed to mitigate cross-image information leakage during the inference phase, meaning it works without needing to retrain the model or change its core structure. The core idea behind FOCUS is to guide the model to concentrate on one image at a time, even when multiple images are present.
FOCUS operates in three main steps:
Visual Masking
First, for each image in a multi-image input, FOCUS creates a “partially masked” version. This involves corrupting all images except the one currently being focused on with random noise. This noise effectively “hides” the irrelevant images, prompting the LVLM to concentrate solely on the single, clean image.
Image-wise Focused Inference
Next, the model performs multiple forward passes. In each pass, only one image is kept clean while others are masked. This generates a specific “logit distribution” (the model’s raw output scores for potential words) for each clean image. Additionally, a “noise-only” reference logit distribution is computed by masking all images, providing a baseline for irrelevant visual content.
Also Read:
- Bridging the Latency Gap: How SpotVLM Enhances Real-time AI with Cloud-Edge Context Transfer
- Advancing AI’s Visual Understanding with Region-Level Context
Contrastive Aggregation
Finally, the logit distributions from the individual focused inferences are combined. To suppress any lingering “leakage” signals from the masked images, the noise-only reference logit is subtracted from each individual image’s logit. This “contrastive” refinement ensures that the final output accurately reflects the model’s understanding of each image independently, while still preserving the original positional context of the images.
The effectiveness of FOCUS has been rigorously tested across various multi-image benchmarks, including Winoground, VisMin, Mantis-Eval, and MuirBench, and on different LVLM families like InternVL3, Qwen2.5-VL, and LLaVA-OneVision. The results consistently show significant performance improvements, particularly in metrics that directly measure multi-image reasoning. For instance, on the VisMin benchmark, FOCUS boosted the Image Score for InternVL3-2B from 18.40 to 50.55, demonstrating its substantial impact, especially on smaller models.
Qualitative examples further highlight FOCUS’s ability to disentangle visual information. Where a baseline model might incorrectly combine elements from different images (e.g., describing “bananas and oranges” when only one is present in the queried image), FOCUS accurately provides image-specific descriptions. This innovative approach offers a practical and generalizable solution for enhancing multi-image reasoning in LVLMs without the need for costly retraining or architectural changes. For more technical details, you can refer to the original research paper: Mitigating Cross-Image Information Leakage in LVLMs for Multi-Image Tasks.
