TLDR: EgoCross is a new benchmark for evaluating AI models’ ability to answer questions about first-person (egocentric) videos. Unlike previous benchmarks focused on daily life, EgoCross includes challenging domains like surgery, industry, extreme sports, and animal perspectives. It reveals that current AI models struggle to generalize to these diverse scenarios, highlighting a need for more robust and adaptable AI in real-world applications. The paper also explores methods like fine-tuning and reinforcement learning to improve performance.
Recent advancements in Artificial Intelligence, particularly in Multimodal Large Language Models (MLLMs), have made significant strides in understanding videos from a first-person perspective, known as egocentric video understanding. This field is crucial for applications like wearable assistants, embodied AI, and human-to-robot learning, as it helps AI understand the world as humans perceive it.
However, a new research paper titled “EgoCross: Benchmarking Multimodal Large Language Models for Cross-Domain Egocentric Video Question Answering” by Yanjun Li, Yuqian Fu, Tianwen Qian, Qi’ao Xu, Silong Dai, Danda Pani Paudel, Luc Van Gool, and Xiaoling Wang, highlights a significant limitation in current AI models. While existing benchmarks for egocentric video question answering (EgocentricQA) primarily focus on common daily activities like cooking or cleaning, real-world scenarios often involve vastly different and more complex environments.
The Challenge of Domain Shift
The core problem identified by the researchers is the ‘domain shift.’ This refers to situations where the visual style and semantic content of a video differ substantially from what the AI models were originally trained on. For instance, an AI trained on cooking videos might struggle to identify specific surgical instruments or understand the rapid movements in extreme sports. This gap means that current MLLMs, whether general-purpose or specialized for egocentric data, often fail to generalize effectively to these unfamiliar domains.
Introducing EgoCross: A New Benchmark
To address this critical gap, the researchers introduce EgoCross, a comprehensive benchmark designed specifically to evaluate how well MLLMs can generalize across different domains in EgocentricQA. EgoCross covers four diverse and challenging domains:
- Surgery: Focusing on precise tool identification and surgical phases.
- Industry: Involving complex workflows, component recognition, and procedural understanding.
- Extreme Sports: Characterized by rapid camera motion, unique environments, and blur.
- Animal Perspective: Introducing non-human motion patterns and camera angles.
The benchmark comprises approximately 1,000 question-answering (QA) pairs across 798 video clips. These QA pairs span four key tasks: prediction (forecasting future actions), recognition (identifying objects, actions, events), localization (pinpointing spatial or temporal locations), and counting (tracking instances over time). Each question is provided in both OpenQA (free-form answer) and CloseQA (multiple-choice) formats for detailed evaluation.
Key Findings and Limitations
Extensive experiments conducted on EgoCross revealed that most existing MLLMs perform poorly in these cross-domain settings. On average, models achieved less than 55% accuracy on CloseQA and below 35% on OpenQA. This is a significant drop compared to their performance on daily-life benchmarks, clearly demonstrating their struggle to adapt to new visual and semantic contexts. Surprisingly, even models specifically designed for egocentric video understanding did not perform significantly better, underscoring the depth of the generalization challenge.
Also Read:
- Assessing Emotional Intelligence in Large Language Models: Introducing MME-Emotion
- AEGIS: A New Benchmark for Detecting Hyper-Realistic AI-Generated Videos
Exploring Solutions and Future Directions
The paper also includes pilot studies exploring potential improvements, such as prompt learning, supervised fine-tuning (SFT), and reinforcement learning (RL). These studies showed that while each method offered some improvement, reinforcement learning yielded the most significant gains, suggesting promising avenues for future research in building more robust and domain-adaptive MLLMs.
The introduction of EgoCross provides a valuable foundation for advancing egocentric video understanding, pushing the boundaries beyond common daily activities towards more realistic and high-impact application scenarios. Researchers and developers can access the data and code for EgoCross to further their work in this area. You can find more details about the research paper here.
