TLDR: This paper details an approach to Visual Question Answering (VQA) for gastrointestinal endoscopy, part of the ImageCLEFmed MEDVQA 2025 Challenge. The researchers used the Florence-2 multimodal foundation model, fine-tuning it with domain-specific augmentations on the Kvasir-VQA dataset. Their method, which combines a powerful vision encoder with a text encoder, successfully generates clinically relevant answers from endoscopic images. The study highlights the potential of large multimodal models in medical VQA, showing accurate results and providing a strong baseline for future work on interpretability, robustness, and clinical integration.
In the realm of medical diagnostics, the accurate interpretation of gastrointestinal (GI) endoscopic images is crucial for the early detection and treatment of diseases. Recent advancements in artificial intelligence, particularly in deep learning, are offering promising solutions to automate this complex analysis. A new research paper explores how Visual Question Answering (VQA) systems can enhance these efforts by linking image understanding with natural language queries to provide actionable clinical insights.
The paper, titled “Multimodal AI for Gastrointestinal Diagnostics: Tackling VQA in MEDVQA-GI 2025,” addresses Subtask 1 of the ImageCLEFmed MEDVQA 2025 Challenge, which focuses specifically on VQA for GI endoscopy. The goal is to develop intelligent systems that can automatically answer clinically relevant questions based on GI images, thereby supporting clinical diagnosis, documentation, and education.
The researchers, Sujata Gaihre, Amir Thapa Magar, Prasuna Pokharel, and Laxmi Tiwari, adopted Florence-2 as the core of their VQA system. Florence-2 is a large-scale multimodal foundation model, meaning it’s designed to understand and process both visual (images) and textual (language) information. This model acts as the backbone, combining a powerful vision encoder to interpret endoscopic images with a text encoder to understand questions and generate clinically relevant answers.
A key aspect of their approach involved using domain-specific augmentations. These are special modifications applied to the training images that preserve important medical features, like mucosal texture or bleeding points, while simulating the variability found in real-world endoscopic procedures. This technique helps the model generalize better and perform accurately on unseen data.
The team utilized the Kvasir-VQA dataset for their experiments. This dataset is a rich collection of approximately 58,849 image-question-answer (IQA) triplets derived from the HyperKvasir image repository, featuring high-resolution GI endoscopy images. Each entry includes an image, its clinical label, a natural language question about diagnostic, anatomical, or procedural aspects, and a concise answer. For efficiency, a stratified 1% subset of this dataset was used for training and evaluation.
The methodology involved fine-tuning the Florence-2 model. This process adapts a pre-trained general-purpose model to a specific task, in this case, medical VQA. The vision part of the model was kept frozen to retain its robust pre-trained features, while the language decoding part was trained to align with the specific linguistic patterns of medical questions and answers. The model was trained to generate free-text answers, moving beyond simple classification-based systems that offer limited expressiveness.
The experiments demonstrated that fine-tuning Florence-2 with these carefully chosen augmentations yielded accurate responses. The model showed improved performance on public and private test sets compared to the validation set, suggesting its ability to generalize. An analysis of performance by question type revealed that the model performed stronger on spatial questions (e.g., those starting with “where”) and binary questions (e.g., “have”), while struggling more with procedural or abstract questions (e.g., “how”). This indicates strengths in visual pattern recognition but highlights areas for improvement in complex clinical reasoning.
The results underscore the significant potential of large multimodal models in medical VQA, providing a strong foundation for future work. The researchers also made their code publicly available, fostering further research and development in this critical area. You can find the code and more details about their work at their GitHub repository.
Also Read:
- SynDiff: Enhancing Medical Image Segmentation with Text-Guided Synthetic Data and Single-Step Diffusion
- EndoControlMag: Enhancing Vascular Visualization in Endoscopic Surgery
Future directions for this research include enhancing model interpretability through visual grounding, incorporating uncertainty handling for unanswerable questions, integrating external medical knowledge to enrich semantic reasoning, and extending the system to handle multi-turn, conversational scenarios. These advancements aim to improve the system’s reliability, transparency, and alignment with real-world clinical workflows, ultimately contributing to better gastrointestinal diagnostics.
