TLDR: A new study identifies “Easy-Options Bias” (EOB) in multiple-choice Visual Question Answering (VQA) benchmarks, where AI models can answer questions without reading them, relying only on visual input and answer options. This bias stems from incorrect options being too visually distinct from the correct one. To fix this, researchers developed “GroundAttack,” a tool that generates challenging, visually plausible negative options, forcing models to genuinely use the question for reasoning and providing a more accurate evaluation of their multimodal understanding.
In the rapidly evolving field of artificial intelligence, Visual Question Answering (VQA) has emerged as a crucial benchmark for evaluating how well models can understand and reason about both visual and linguistic information. VQA tasks typically involve showing a model an image or video and asking a natural language question, expecting it to select the correct answer from a set of options. The assumption is that a high score indicates a model’s ability to integrate visual content with question semantics.
However, a recent study by Hao Zhang, Chen Li, and Basura Fernando has uncovered a significant flaw in many popular multiple-choice VQA benchmarks, which they term the “Easy-Options Bias” (EOB). This bias allows advanced vision-language models (VLMs) to correctly answer questions without actually needing to process the question itself. Instead, these models can often infer the correct answer by simply looking at the visual input and the provided answer options.
The researchers observed this phenomenon across several well-known VQA benchmarks, including MMStar, RealWorldQA, SEED-Bench, NExT-QA, STAR benchmark, and Video-MME. They found that VLMs, when given only the visual input and the answer choices (V+O settings), still achieved surprisingly high accuracies, often just slightly lower than when they also had access to the question (V+Q+O settings). This suggests that the negative (incorrect) answer options in these benchmarks are often too easily distinguishable from the correct answer based on visual cues alone, creating a “shortcut” for the models.
To understand the root cause of this bias, the team conducted grounding experiments using CLIP, a pre-trained vision-language alignment model. Their findings revealed a consistent pattern: the correct answer options consistently showed a stronger visual-text alignment with the image or video content compared to the incorrect distractors. This “visual relevance imbalance” means that the correct answer is not just semantically appropriate but also more visually obvious, allowing models to bypass the complex reasoning that VQA tasks are designed to test.
To address this critical issue, the researchers introduced a practical toolkit called GroundAttack. This innovative tool automatically generates “hard negative options” that are designed to be as visually plausible and semantically confusing as the correct answer. GroundAttack works by replacing only the incorrect answer choices in existing VQA datasets, while keeping the original vision, question, and correct answer intact.
The GroundAttack toolkit operates through three main components: a Captioner, which converts visual content into detailed textual descriptions; a Distractor, which generates a large pool of plausible, yet incorrect, candidate options; and a Selector, which then intelligently picks the most challenging and diverse negative options from this pool. By using large pre-trained models for these components, GroundAttack minimizes the need for manual effort and ensures consistency.
Experiments on the NExT-QA and MMStar datasets demonstrated the effectiveness of GroundAttack. When VLMs were evaluated on datasets augmented with GroundAttack-generated negative options, their accuracies dropped significantly, approaching random guessing levels in the (V+O) setting. This indicates that GroundAttack successfully mitigates the Easy-Options Bias, forcing models to rely on the question for accurate answers and thus providing a more realistic and robust evaluation of their true multimodal reasoning capabilities.
Also Read:
- Understanding Vulnerabilities in AI Explanations for Visual Questions
- Assessing AI’s Spatial Awareness: A Deep Dive into Image Rotation Challenges
The study highlights the urgent need for a re-evaluation of how VQA benchmarks are constructed. Future benchmarks must ensure that questions are indispensable for arriving at the correct answer, preventing models from exploiting superficial correlations or visual obviousness. This work paves the way for developing more diagnostic and bias-resistant tools to truly measure multimodal understanding in advanced vision-language systems. You can read the full research paper here: Mitigating Easy Option Bias in Multiple-Choice Question Answering.
