TLDR: OmniBrainBench is the first comprehensive multimodal benchmark for evaluating AI models (MLLMs) in brain imaging analysis. It features 15 imaging modalities and 15 multi-stage clinical tasks, simulating real-world clinical workflows. Evaluation of 24 MLLMs revealed that proprietary models generally outperform others but all AI models significantly lag behind human physicians, especially in complex reasoning tasks, highlighting a critical gap between visual perception and medical comprehension.
Brain imaging analysis is a critical component in the diagnosis and treatment of various brain disorders. With the rise of multimodal large language models (MLLMs), there’s a growing potential for AI to assist in this complex field. However, a significant challenge has been the lack of comprehensive benchmarks to truly assess how well these AI models understand and process brain imaging data across the full spectrum of clinical tasks.
Existing benchmarks often fall short by covering only a limited number of imaging modalities or focusing on very specific, coarse-grained pathological descriptions. This narrow scope prevents a thorough evaluation of MLLMs as they would be used in real-world clinical settings, where diverse imaging types and multi-stage diagnostic processes are common.
To address this crucial gap, researchers have introduced OmniBrainBench, the first comprehensive multimodal visual question-answering (VQA) benchmark specifically designed for brain imaging analysis. This new benchmark aims to provide a robust framework for evaluating the multimodal comprehension capabilities of MLLMs.
What Makes OmniBrainBench Unique?
OmniBrainBench stands out due to its extensive coverage and clinical relevance. It incorporates 15 distinct brain imaging modalities, gathered from 30 verified medical sources. This rich dataset includes 9,527 validated VQA pairs and a massive collection of 31,706 images. The modalities range from common ones like CT and MRI to more specialized types such as PET, SPECT, DWI, FLAIR, and fMRI, covering structural, functional, and molecular neuroimaging.
Beyond just diverse imaging types, OmniBrainBench simulates real clinical workflows. It encompasses 15 multi-stage clinical tasks, all rigorously validated by a professional radiologist. These tasks are grouped into five specialized clinical phases:
- Anatomical and Imaging Assessment (AIA)
- Lesion Identification and Localization (LIL)
- Diagnostic Synthesis and Causal Reasoning (DSCR)
- Prognostic Judgment and Risk Forecasting (PJRF)
- Therapeutic Cycle Management (TCM)
This structure allows for a detailed evaluation of MLLMs across the entire clinical continuum, from basic anatomical recognition to complex diagnostic synthesis, prognostic judgment, and therapeutic cycle management.
Evaluating State-of-the-Art AI Models
The researchers evaluated 24 state-of-the-art MLLMs on OmniBrainBench, including open-source, medical-specific, and proprietary models. Human clinician performance was used as a reference point to highlight the gaps between AI and expert medical reasoning.
The experiments revealed several key insights:
- Proprietary MLLMs, such as GPT-5 and Gemini-2.5-Pro, generally outperformed open-source and medical-specific models. Gemini-2.5-Pro achieved the highest overall score, excelling in several subtasks.
- Despite the strong performance of leading AI models, a substantial gap remains between MLLMs and human physicians. The highest-performing AI model lagged behind the physician’s average accuracy by approximately 24.77%.
- Medical-specific MLLMs showed varied performance, with some like HuatuoGPT-V-34B being highly competitive, while others displayed significantly lower scores.
- Open-source MLLMs generally trailed in overall performance but demonstrated specific strengths in certain tasks, suggesting potential for targeted optimization.
- The benchmark highlighted significant variations in task difficulty for MLLMs. Models performed well in tasks like prognostic factor analysis and clinical sign prediction but struggled considerably with more complex tasks such as risk stratification and preoperative assessment. This indicates a gap between visual perception and deeper medical comprehension and reasoning.
Also Read:
- Advancing Medical Image Analysis with Adaptable Foundation Models
- Advancing Medical Visual Question Answering with Cross-Mamba Interaction and Multi-Task Learning
The Path Forward
OmniBrainBench sets a new standard for evaluating and advancing MLLMs in brain imaging analysis. It not only highlights the current capabilities of AI models but also critically exposes their limitations, particularly in complex preoperative tasks and nuanced clinical scenarios. The findings underscore the urgent need for further advancements in domain adaptation and prompt engineering to bridge the performance gap between AI and expert clinical reasoning.
This benchmark is expected to catalyze progress in developing clinically viable AI solutions for brain imaging, serving as a vital experimental arena to accurately assess MLLM performance and reduce costs before real-world deployments. However, it’s important to remember that while comprehensive, OmniBrainBench is a preliminary step and cannot replace final clinical evaluation for safety.
