TLDR: A new benchmark called EGOILLUSION has been introduced to evaluate how Multimodal Large Language Models (MLLMs) hallucinate when processing egocentric (first-person) videos. The benchmark includes 1,400 videos and 8,000 human-annotated questions designed to trigger hallucinations in both visual and auditory cues. Evaluations of ten MLLMs, including advanced models like GPT-4o and Gemini, showed significant challenges, with the best model achieving only 59% accuracy, highlighting a substantial gap compared to human performance and indicating that MLLMs struggle particularly with reasoning tasks and audio cues in egocentric video understanding.
Multimodal Large Language Models (MLLMs) have shown impressive capabilities in understanding and processing various forms of data, including complex videos. These models are designed to interpret both visual and auditory information, enabling them to perform tasks like identifying objects, understanding actions, and even reasoning about events. However, despite their advancements, MLLMs are not perfect and often generate responses that, while seemingly coherent, are factually incorrect. This phenomenon is known as ‘hallucination’.
A new research paper titled “EGOILLUSION : Benchmarking Hallucinations in Egocentric Video Understanding” introduces a groundbreaking benchmark specifically designed to evaluate these hallucinations in a unique and challenging context: egocentric videos. Unlike traditional third-person videos captured from a spectator’s viewpoint, egocentric videos are recorded from a first-person perspective, typically from wearable devices. This viewpoint presents distinct challenges, such as dynamic object states, occlusions from hand movements, and a rich array of multi-sensory information, including actions, object appearances, and ambient sounds.
The EGOILLUSION benchmark, developed by researchers Ashish Seth, Utkarsh Tyagi, Ramaneswaran Selvakumar, Nishit Anand, Sonal Kumar, Sreyan Ghosh, Ramani Duraiswami, Chirag Agarwal, and Dinesh Manocha, is the first of its kind to focus on hallucinations in egocentric video understanding. It comprises 1,400 diverse egocentric videos, ranging from 30 seconds to over 5 minutes in length, paired with an extensive set of 8,000 human-annotated question-answer pairs. These questions are carefully crafted to trigger hallucinations in MLLMs, covering both visual and auditory cues, and include both open-ended and closed-ended formats.
The creation of EGOILLUSION involved a meticulous data collection and filtering process. Videos were sourced from various egocentric datasets like Ego4D-HCap, EgoSeg, EPIC-KITCHENS, and Trek-150, ensuring a wide range of visual scenarios such as cooking, painting, and navigating urban environments. A manual filtering step ensured that only visually and temporally rich scenarios, crucial for generating complex queries, were included. To enhance the video narrations, an automated pipeline was developed to identify both active (interacted with) and non-active (background) objects using advanced AI models like GPT-4o, and to incorporate environmental sounds using Qwen2-Audio. This enriched narration provides a more comprehensive context for generating questions.
The benchmark features six core egocentric video-language tasks: Episodic Information Reasoning (EIR), Temporal Reasoning (TR), Human-Object Interaction (HOI), Visual Object Identification (VOI), Object State Change Detection (OSCD), and Audio Event Recognition (AER). These tasks are designed to assess either a model’s perception (interpreting multi-sensory inputs) or its reasoning capabilities (inferring knowledge, explaining causality). To induce hallucinations, the annotators employed specific strategies: prompt injection (misleading instructions), adversarial sampling (replacing active objects with plausible non-active ones), and manipulating temporal order (altering event sequences).
The researchers evaluated ten state-of-the-art MLLMs, including popular models like GPT-4o and Gemini-Pro, on the EGOILLUSION benchmark. The results revealed significant challenges for current MLLMs. The best-performing model, Gemini-Pro, achieved an accuracy of only 59.4%, which is a substantial gap compared to human performance of 86.1%. Interestingly, the study found only a minimal performance difference between open-source and closed-source models, as well as between smaller and larger MLLMs, suggesting that simply scaling model size does not effectively mitigate hallucinations in this domain.
Further analysis showed that MLLMs hallucinate more frequently on complex reasoning tasks (like Temporal Reasoning and Episodic Information Reasoning) compared to perception-based tasks (like Visual Object Identification and Audio Event Recognition). Models also struggled consistently across all hallucination-inducing strategies, with a notable difficulty in understanding chronological ordering and susceptibility to misleading prompts. A key finding was that MLLMs struggle significantly with both visual and audio modalities, but particularly fail to leverage audio cues effectively, often relying on language priors instead, which leads to more hallucinated responses.
Also Read:
- Assessing AI’s Ability to Mimic Low Vision Perception
- Unpacking Object Reasoning: ORBIT Benchmark Exposes VLM Limitations
Error analysis revealed a common ‘Yes/No bias’ in closed-ended questions, where models showed a strong inclination towards affirmative responses in hallucinated outputs. Perception errors were identified as the primary source of mistakes, followed by logical and procedural errors, highlighting the critical need for improved perceptual understanding in egocentric video tasks. The EGOILLUSION benchmark serves as a crucial tool for identifying the limitations of current MLLMs in egocentric video understanding and provides a clear roadmap for future research aimed at developing more robust and less hallucinatory models. You can find more details about this research paper at https://arxiv.org/pdf/2508.12687.
