TLDR: MVU-Eval is the first comprehensive benchmark designed to evaluate Multimodal Large Language Models (MLLMs) on their ability to understand and reason across multiple videos. It addresses a critical gap in existing evaluations, which primarily focus on single-video understanding. Comprising 1,824 question-answer pairs and 4,959 videos, MVU-Eval assesses eight core perception and reasoning competencies relevant to real-world applications like autonomous driving and sports analytics. Initial evaluations reveal significant limitations in current MLLMs’ multi-video understanding capabilities, highlighting the need for improved architectures and training strategies.
The field of Artificial Intelligence has seen remarkable advancements with Multimodal Large Language Models (MLLMs), which extend the capabilities of traditional language models to include visual information like images and videos. While these models have shown promise in understanding single videos, a significant gap has existed in evaluating their ability to comprehend and reason across multiple video streams simultaneously. This limitation is particularly evident in real-world scenarios such as analyzing sports from various camera angles or synthesizing information from multiple sensors in autonomous driving systems.
To address this crucial need, a new benchmark called MVU-Eval has been introduced. This is the first comprehensive evaluation framework specifically designed for Multi-Video Understanding in MLLMs. The benchmark aims to provide a robust method for assessing how well these advanced AI models can process and interpret information from several videos at once.
What MVU-Eval Measures
MVU-Eval is built upon 1,824 carefully crafted question-answer pairs, drawing from a vast collection of 4,959 videos across diverse domains including daily life, movies, gaming, and autonomous driving. It evaluates eight core competencies, divided into two main categories: fundamental perception tasks and higher-order reasoning tasks.
The **Perception** tasks focus on the model’s ability to accurately ‘see’ and ‘identify’ specific content across multiple videos. These include:
- Object Recognition (OR): Identifying and tracking the same objects across different video sequences.
- Spatial Understanding (SU): Comprehending the spatial layout from complementary camera angles.
- Counting: Accurately counting transient objects appearing across asynchronous videos.
- Comparison: Differentiating features and attributes between multiple videos, including identifying replacements, removals, or additions.
The **Reasoning** tasks assess the model’s capacity to analyze and infer meaningful conclusions beyond simple visual recognition, requiring more complex cognitive functions. These include:
- Knowledge-Intensive Reasoning (KIR): Integrating domain-specific knowledge (e.g., sports rules) with multi-video evidence to resolve ambiguities.
- In-Context Learning (ICL): Adapting reasoning strategies from limited examples to new cross-video scenarios.
- Retrieval-Augmented Generation (RAG): Identifying and synthesizing relevant visual evidence from potentially redundant multi-video inputs.
- Temporal Reasoning (TR): Benchmarking the ability to align discontinuous events chronologically across videos with varying timelines.
How MVU-Eval Was Built
The creation of MVU-Eval involved a meticulous data collection process that combined automated generation with rigorous human verification. Video pairs were sampled from various open-source datasets based on specific rules to ensure inter-video relationships. Question-answer pairs were then generated, followed by multiple rounds of quality control. This included removing ‘leaked information’ (where questions could be answered without watching the videos) and filtering out overly easy questions using existing MLLMs. Finally, human annotators reviewed each question-answer pair and corresponding videos to ensure utility, challenge, and correctness, discarding a significant portion of the initial data to maintain high quality.
Key Findings and Insights
Extensive evaluations of both open-source and closed-source MLLMs on MVU-Eval revealed several important observations:
- The benchmark is highly challenging. Even top-performing closed-source models like Gemini 2.5 Pro achieved an accuracy of only 56.6%, significantly lower than human performance. Most open-source MLLMs scored below 50%.
- Model capabilities vary significantly across different subtasks, indicating an imbalance in their current understanding.
- Generally, larger models tend to perform better, demonstrating the importance of model scale.
- Supporting longer contexts, such as more frames and higher resolutions, is crucial for MLLMs, though excessive input can sometimes degrade performance.
- The presence and quality of visual information are critical; performance consistently drops when less visual data is provided.
- The format in which multiple videos are input to the model can also significantly impact performance.
Analysis of failure cases showed that MLLMs struggle with interpreting object status or function in perception tasks and face difficulties with domain-specific knowledge, filtering irrelevant information, and understanding complex temporal and causal relationships in reasoning tasks. For example, models might identify an object but fail to understand if it’s being actively used, or struggle to integrate spatial information from different camera angles.
Also Read:
- IndicVisionBench: A New Frontier for Evaluating AI’s Cultural and Multilingual Understanding in India
- SPATIALTHINKER: Advancing 3D Spatial Understanding in Multimodal AI Models
Future Directions
The MVU-Eval benchmark highlights several promising areas for future research. These include developing better methods for cross-video visual alignment, enhancing spatial understanding across multiple viewpoints, improving temporal reasoning in asynchronous multi-video scenarios, and exploring scalable multi-modal fusion strategies to handle a greater number of videos without overwhelming models. The benchmark will be made publicly available to foster further research and development in this critical area of AI. For more details, you can refer to the full research paper: MVU-Eval: Towards Multi-Video Understanding Evaluation for Multimodal LLMs.
