TLDR: OmniVideoBench is a new, large-scale benchmark introduced by the NJU-LINK Team to evaluate multimodal large language models (MLLMs) on their synergistic audio-visual understanding. It features 1,000 high-quality, manually verified question-answer pairs from 628 diverse real-world videos (seconds to 30 minutes), covering 13 task types with step-by-step reasoning traces. Evaluations reveal a significant gap between MLLM performance and human reasoning, with models struggling particularly with long videos and music-dominated audio, highlighting the need for improved cross-modal reasoning and long-context processing.
The rapid advancements in multimodal large language models (MLLMs) have opened new frontiers in how artificial intelligence understands and interacts with video content. However, a significant challenge has been the lack of comprehensive benchmarks that truly evaluate an MLLM’s ability to synergistically reason across both audio and visual modalities. Often, existing evaluation tools either overlook one modality or integrate them in ways that lack logical consistency.
To address this critical gap, the NJU-LINK Team has introduced OmniVideoBench, a meticulously designed, large-scale benchmark aimed at assessing the synergistic audio-visual understanding capabilities of MLLMs. This new benchmark places a strong emphasis on how well models can use complementary information from both audio and visual streams and maintain logical consistency in their reasoning.
What Makes OmniVideoBench Unique?
OmniVideoBench stands out with its impressive scale and rigorous design. It comprises 1,000 high-quality question-answer (QA) pairs, each accompanied by detailed, step-by-step reasoning traces. These QA pairs are derived from 628 diverse videos, varying in length from just a few seconds to a full 30 minutes. Every question and answer has been manually verified to ensure complete correctness and uniqueness, providing a robust foundation for evaluation.
The benchmark covers 13 carefully designed question types, capturing the essential challenges of video understanding. These include tasks requiring temporal reasoning (understanding events over time), spatial localization (identifying where things are), counting, causal inference (understanding cause-and-effect), summarization, and more. This broad range ensures a holistic assessment of an MLLM’s capabilities.
Rigorous Data Collection and Quality Assurance
The videos in OmniVideoBench are sourced from real-world platforms like YouTube and Bilibili, ensuring rich and authentic audiovisual content. The collection process prioritized diversity in both video type and duration. Videos are categorized into eight major classes (e.g., Vlog, News, Sports, Documentary) and further subdivided into 68 fine-grained subcategories. Video lengths range from short clips to extended 30-minute segments, pushing models to handle varying temporal scales.
To guarantee the highest quality, a multi-stage annotation and refinement pipeline was employed. Initially, approximately 2,500 multiple-choice QA pairs were manually created. These questions were designed to necessitate audio-visual reasoning, with answers being both correct and unique. Crucially, questions were filtered to remove those solvable by a single modality (vision or audio alone) or by common-sense knowledge or textual cues. This rigorous process, involving advanced MLLMs and large language models for filtering, ensured that the final 1,000 QA pairs genuinely test synergistic audio-visual understanding.
Key Findings from Model Evaluations
Evaluations of various MLLMs on OmniVideoBench have revealed several insightful findings, highlighting the current limitations of these models:
-
Significant Performance Gap: There’s a pronounced gap between current model performance and human reasoning abilities. Open-source models, in particular, lag significantly behind their closed-source counterparts, with the best-performing model, Gemini-2.5-Pro, achieving an accuracy of 58.90%.
-
Challenges with Long Videos: While some leading models show relatively robust performance on longer videos, many MLLMs still struggle with understanding content that spans extended durations, indicating room for improvement in long-term temporal reasoning.
-
Difficulty with Music Signals: Models exhibit lower accuracy when dealing with videos dominated by music signals compared to those with speech or ambient sounds. This suggests that current MLLMs have limited capability in translating abstract emotional and atmospheric information from music into effective reasoning.
-
Varying Task Performance: Performance differs significantly across task types. Tasks requiring background and music understanding, which rely on low-semantic acoustic cues, proved to be the most challenging. In contrast, tasks like relationship reasoning and summarization, which often depend more on explicit language and visual observation, showed higher accuracy.
-
Audio’s Irreplaceable Role: While Automatic Speech Recognition (ASR) transcripts can improve performance for visual-only models, genuine audio comprehension remains irreplaceable, especially for tasks requiring deeper understanding of sounds and music.
-
Impact of Frame Sampling: Increasing the number of video frames provided to models consistently improved accuracy, particularly for longer videos, underscoring the importance of dense temporal information for robust audiovisual reasoning.
Also Read:
- Ro-Bench: A New Standard for Testing Video AI’s Resilience to Manipulated Content
- Bridging Cultural Gaps: A New Framework to Evaluate AI’s Understanding of Asian Contexts
The Future of Multimodal AI
OmniVideoBench serves as a crucial tool for advancing the field of multimodal AI. By exposing the current weaknesses of MLLMs in synergistic audio-visual reasoning, long-form temporal understanding, and music comprehension, it provides a clear roadmap for future research. The release of OmniVideoBench is expected to foster the development of MLLMs with stronger and more generalizable reasoning capabilities, bringing us closer to human-level multimodal understanding.
