TLDR: A study evaluated GPT-4o, GPT-5-nano, GPT-5-mini, and GPT-5 on a brain tumor visual question answering (VQA) benchmark using MRI scans. GPT-5-mini showed the highest overall accuracy (44.19%), but performance across all models was moderate and not yet suitable for clinical use. The research highlights the potential of LLMs in medical imaging but emphasizes the need for further development, including human expert comparisons and domain-specific fine-tuning, to enhance their capabilities for clinical application.
Understanding brain tumors accurately from Magnetic Resonance Imaging (MRI) scans is crucial for effective treatment planning in neuro-oncology. This complex task typically requires highly specialized neuroradiological expertise, and even then, interpretations can vary due to differences in experience or fatigue. This is where artificial intelligence, particularly large language models (LLMs), shows promise in enhancing diagnostic efficiency and standardization.
A recent study, titled Performance of GPT-5 in Brain Tumor MRI Reasoning, explored the capabilities of various GPT-5 family models, including GPT-4o, GPT-5-nano, GPT-5-mini, and GPT-5, in interpreting brain tumor MRI scans. The research focused on a method called Visual Question Answering (VQA), which combines image interpretation with natural language reasoning to answer clinically relevant questions about medical images.
The researchers developed a specialized benchmark using data from three Brain Tumor Segmentation (BraTS) datasets, covering glioblastoma (GLI), meningioma (MEN), and brain metastases (MET). For each case, multi-sequence MRI images were processed into standardized triplanar mosaics. Alongside these images, structured clinical features, such as lesion location, enhancement patterns, and edema, were converted into a series of yes/no and multiple-choice VQA questions. This meticulous preparation ensured that each model was evaluated on identical, clinically relevant prompts, allowing for direct comparison of their performance.
The models were tested in a zero-shot chain-of-thought setting, meaning they had to reason through the problem step-by-step without prior specific training on this dataset. The results showed that GPT-5-mini achieved the highest macro-average accuracy at 44.19%, closely followed by GPT-5 at 43.71%. GPT-4o came in third with 41.49%, and GPT-5-nano had the lowest accuracy at 35.85%. It’s important to note that no single model consistently outperformed others across all tumor types; for instance, GPT-4o performed best on glioblastoma cases, while GPT-5 excelled with metastases and meningiomas.
The study’s findings suggest that while GPT-5 family models can achieve moderate accuracy in structured neuro-oncological VQA tasks, their current performance level is not yet acceptable for direct clinical use. The small difference in accuracy between GPT-5 and GPT-5-mini indicates that factors beyond just model size might influence performance in this specific medical domain. It’s hypothesized that smaller models might be more conservative in their decision-making, potentially reducing susceptibility to misleading answer choices.
The researchers also discussed limitations, including the use of automated reports for feature extraction, the absence of a human expert comparison to gauge real-world clinical utility, and the sole reliance on accuracy as a metric. The zero-shot chain-of-thought prompting, while encouraging structured reasoning, might also lead to overinterpretation of ambiguous features. Future research aims to address these limitations by incorporating human expert comparisons, developing open-form VQA tasks for more nuanced evaluations, and exploring domain-specific fine-tuning to improve performance further.
Also Read:
- Generative AI’s Impact on Medical Imaging: From Foundations to Clinical Use
- Large Language Models Know Clinical Facts, But Struggle to Reason with Them
In conclusion, this study provides valuable insights into the current capabilities of advanced large language models in a critical medical application. While demonstrating meaningful, albeit limited, abilities in brain tumor MRI reasoning, it also highlights the significant work still needed to bring these technologies to a level suitable for practical clinical integration. The benchmark developed in this research serves as a crucial tool for future advancements in this field.
