TLDR: SciVideoBench is a new benchmark designed to evaluate advanced video reasoning in scientific contexts for Large Multimodal Models (LMMs). It features 1,000 complex multiple-choice questions derived from research-level experimental videos across over 25 scientific subjects. Evaluations reveal significant performance gaps in current state-of-the-art LMMs, particularly in quantitative reasoning, highlighting the need for further development in AI’s ability to understand and reason about scientific experiments.
Large Multimodal Models (LMMs) have made incredible strides in various areas, from conversation to code generation and image understanding. However, when it comes to understanding complex videos, especially in the nuanced world of scientific research, these advanced AI systems still face significant hurdles. Current video benchmarks often focus on general scenarios and simpler reasoning tasks, which means they don’t fully test the sophisticated cognitive skills needed for scientific discovery.
To bridge this crucial gap, researchers have introduced SciVideoBench, a groundbreaking benchmark specifically designed to rigorously evaluate advanced video reasoning in scientific contexts. This new benchmark features 1,000 meticulously crafted multiple-choice questions. These questions are drawn from cutting-edge scientific experimental videos, spanning over 25 specialized academic subjects, and have been carefully verified by a semi-automatic system.
What makes SciVideoBench unique is its demand for sophisticated domain-specific knowledge, precise spatiotemporal perception (understanding what’s happening where and when), and intricate logical reasoning. It truly challenges the higher-order cognitive abilities of LMMs, pushing them beyond simple recognition tasks.
The creation of SciVideoBench involved a multi-stage, agent-human collaborative pipeline. Videos were sourced from the Journal of Visualized Experiments (JoVE), a peer-reviewed platform known for publishing high-quality methodological videos across various scientific disciplines. Each video comes with a peer-reviewed manuscript and synchronized audio narration, providing a rich, multi-modal context for question generation and answer verification. The benchmark focuses on four foundational domains: physics, chemistry, biology, and medicine.
SciVideoBench categorizes its questions into three distinct types:
Conceptual Reasoning
These questions explore the mechanisms, protocols, and scientific principles behind the operations observed in the experiment, inferred from visual and operational cues.
Hypothetical Reasoning
This type focuses on specific experimental operations that are critical to the experiment’s motivation or outcome. It often involves ‘what-if’ analyses, hypothesized errors, or experimental control logic, requiring expert-level knowledge and accurate visual perception.
Also Read:
- Unveiling AI’s Scientific Discovery Prowess: A New Benchmark for Language Models
- Introducing FinMR: A New Benchmark for Advanced Financial AI Reasoning
Quantitative Reasoning
These questions demand numerical perception, reasoning, and calculation, requiring models to extract informative values directly from the video before performing complex computations.
Initial evaluations of both proprietary and open-source LMMs on SciVideoBench have revealed consistently low accuracy. Even state-of-the-art models like Gemini 2.5 Pro and Qwen2.5-VL show significant performance deficits, indicating substantial room for improvement in video reasoning capabilities. For instance, vision-blind baselines, which only receive textual content without video, perform barely above random chance, underscoring the indispensable role of visual information.
Proprietary models generally outperform their open-source counterparts, with Gemini 2.5 Pro achieving the highest overall accuracy. The gap is particularly wide in Quantitative Reasoning, which consistently proves to be the most challenging category across all models. Interestingly, Chain-of-Thought (CoT) prompting, which encourages models to explain their reasoning step-by-step, significantly boosts performance for proprietary models, especially in quantitative tasks. For some open-source models, CoT helps quantitative reasoning but can sometimes hinder conceptual and hypothetical reasoning, suggesting a need for more robust long-form causal explanations.
Detailed analyses of errors point to common issues such as incorrect visual perception, inaccurate reasoning progress, and a lack of domain knowledge. Many wrong responses stem from a combination of these factors, highlighting the complexity of scientific video understanding for current AI systems.
SciVideoBench serves as a rigorous testbed for evaluating current video reasoning abilities and acts as a catalyst for innovation, fostering the development of highly capable AI co-scientists that can accelerate future scientific discovery. For more details, you can read the full research paper here.
