TLDR: This paper evaluates the “thinking mode” of two leading Multimodal Large Language Models (MLLMs), Seed1.5-VL and Gemini-2.5-Flash, in four visual medical tasks using VQA-RAD and ROCOv2 datasets. The study found that while activating the thinking mode generally improves performance, the gains are marginal and only become noticeable with increased task complexity. Performance on highly complex medical tasks remains suboptimal, and the thinking mode can also reduce model output consistency and increase latency. The research highlights the need for domain-specific medical data and advanced methods for medical knowledge integration.
Recent advancements in Artificial Intelligence have introduced a new breed of Multimodal Large Language Models (MLLMs) that possess a unique feature: an explicit ‘thinking mode’. This capability allows these models to engage in a step-by-step internal deliberation process before generating a final response, a stark contrast to their standard ‘non-thinking mode’. This research paper, titled Benchmarking the Thinking Mode of Multimodal Large Language Models in Clinical Tasks, rigorously evaluates how these enhanced reasoning processes impact model performance and reliability, particularly in critical clinical tasks.
The study, conducted by a team of researchers including Jindong Hong, Tianjie Chen, Lingjie Luo, Chuanyang Zheng, Ting Xu, Haibao Yu, Jianing Qiu, Qianzhong Chen, Suning Huang, Yan Xu, Yong Gui, Yijun He, and Jiankai Sun, focuses on two prominent MLLMs: Seed1.5-VL and Gemini-2.5-Flash. The goal was to understand the extent to which activating the thinking mode improves performance in visual-language medical tasks, especially complex ones, and to identify specific tasks where this mode offers the most significant advantages.
Evaluating Performance Across Medical Tasks
To assess the models, the researchers utilized two publicly available datasets: VQA-RAD for visual question answering in radiology and ROCOv2 for medical image captioning and concept detection. Four distinct medical tasks were devised:
- Close-ended Visual Question Answering (VQA): Involving ‘yes/no’ questions based on radiological images.
- Open-ended VQA: Requiring models to generate concise free-text answers to questions about medical images.
- Concept Detection: Tasking the model to predict relevant medical concepts (e.g., UMLS CUIs, modalities, anatomical regions) directly from images.
- Caption Prediction: Demanding a full interpretation of the image and generation of a coherent medical caption.
The findings revealed several key insights. Firstly, model performance consistently decreased as task complexity increased, with scores progressively dropping from close-ended VQA to open-ended VQA, concept detection, and caption prediction. This suggests that while MLLMs are capable, their current capabilities are stretched by more intricate medical scenarios.
The Impact of Thinking Mode
A central finding was that the improvement from activating the thinking mode remained marginal compared to the standard non-thinking mode for the majority of tasks. Its advantages only became noticeable when the complexity of medical tasks increased. For instance, in VQA tasks, the thinking mode showed limited benefits, with Gemini-2.5-Flash even showing a slight decline in open-ended VQA. However, for image captioning tasks, the thinking mode demonstrated slightly clearer advantages.
Despite these improvements, the overall performance of both leading MLLMs on highly complex medical tasks remained suboptimal. Except for close-ended VQA, the average scores for other tasks were often close to or below 50%, indicating a significant gap in meeting usability standards for complex clinical applications. This highlights a crucial need for domain-specific training or fine-tuning, as well as more advanced methods for integrating medical knowledge.
Consistency and Latency Considerations
Beyond raw performance, the study also examined model output consistency and speed. It was found that the thinking mode generally resulted in lower model output consistency compared to the non-thinking mode. Gemini-2.5-Flash, in particular, showed a substantial 8.8% drop in consistency when thinking mode was active, suggesting that the internal reasoning processes might introduce more randomness. Seed1.5-VL was more robust in this aspect, with only a marginal drop.
Regarding speed, activating the thinking mode significantly increased inference latency. For open-ended VQA, the thinking mode introduced a considerable delay with very little, or even a negative, impact on performance. This phenomenon, often referred to as the ‘overthinking problem,’ suggests a trade-off between reasoning depth and practical usability in time-sensitive clinical environments.
Also Read:
- Advancing Vision-Language Models with Multi-Prompt Learning
- Unpacking How AI Models Resolve Conflicting Information from Images and Text
Future Directions
The research acknowledges several limitations, including the exclusive use of radiology image datasets, which may limit the generalizability of the findings to other medical imaging modalities. The lack of transparent training data documentation for the evaluated models also raises concerns about potential data leakage. Furthermore, while LLM-based evaluation was validated, the study emphasizes the ongoing need for broader human expert clinical assessments.
In conclusion, while the thinking mode is a valuable enhancement for MLLMs, its current advantages in medical tasks are below expectations. It is not a complete solution for achieving clinical usability with general MLLMs, especially in highly complex visual medical scenarios like image interpretation. The study underscores the importance of understanding task complexity before deploying MLLMs and points to the necessity of integrating domain-specific medical data and advanced multimodal learning methods to truly unlock their potential in healthcare.
