TLDR: AMANDA is a training-free AI framework that improves Medical Visual Question Answering (Med-VQA) by using specialized agents to augment medical knowledge. It addresses limitations in existing models by performing deep visual analysis through question decomposition (intrinsic knowledge) and integrating external medical facts from knowledge graphs (extrinsic knowledge). This leads to significantly better performance and reduced medical “hallucinations” in low-data settings, making AI diagnoses more reliable.
Medical Visual Question Answering (Med-VQA) systems, which allow AI to answer natural language questions about medical images, hold immense potential for assisting healthcare professionals. However, a significant challenge arises when these systems are deployed in settings where there isn’t a lot of labeled medical data available. Existing Medical Multimodal Large Language Models (Med-MLLMs) often struggle in such “low-resource” environments because they face two main reasoning limitations.
The first limitation is an “intrinsic reasoning bottleneck.” Current Med-MLLMs tend to look at images from a general perspective, often missing the subtle, fine-grained details of pathological features that are crucial for accurate diagnoses. Unlike human clinicians who who perform an iterative process of questioning and examination, these AI models typically make a single-step inference, leading to superficial analyses.
The second is an “extrinsic reasoning bottleneck.” While Med-MLLMs have some basic medical knowledge from their training, these models are typically static and can’t easily access or incorporate new, specialized medical information. This can lead to them generating “hallucinations”—plausible but factually incorrect responses—which are highly risky in medical diagnosis.
To overcome these critical issues, researchers have introduced a novel framework called AMANDA (Agentic Medical Knowledge Augmentation). This training-free system enhances the reasoning capabilities of Med-MLLMs by augmenting their medical knowledge using specialized AI agents. You can find the full research paper here: AMANDA: Agentic Medical Knowledge Augmentation for Data-Efficient Medical Visual Question Answering.
How AMANDA Works: A Collaborative Agent System
AMANDA operates through a collaborative network of five specialized AI agents: the Perceiver, Reasoner, Evaluator, Explorer, and Retriever. These agents work together in an iterative process to refine answers and ensure comprehensive and reliable medical reasoning.
The process begins with the Perceiver agent, which uses a Med-MLLM to analyze a medical image and generate a detailed medical caption along with an initial answer to the main question. This sets the foundation for understanding the image’s key features.
Next, the Reasoner agent synthesizes all available information to produce a refined answer. The Evaluator then assesses the confidence of this answer. If the confidence is low, it signals that more knowledge augmentation is needed.
This is where AMANDA’s core knowledge augmentation strategies come into play:
Intrinsic Medical Knowledge Augmentation (Explorer Agent)
To address the intrinsic reasoning bottleneck, the Explorer agent employs a “coarse-to-fine” question decomposition strategy. Inspired by how clinicians approach a diagnosis, this agent breaks down the original complex question into a series of more focused sub-questions. These sub-questions progress from general observations (e.g., “What is the overall appearance of the image?”) to specific anatomical analyses (e.g., “What is the appearance and position of the cardiac silhouette?”) and finally to detailed findings (e.g., “Are there any infiltrates or masses in the lower right lung field?”). By answering these sub-questions, the system performs a deeper, more thorough visual analysis, uncovering crucial diagnostic details that might otherwise be missed.
Extrinsic Medical Knowledge Augmentation (Retriever Agent)
To tackle the extrinsic reasoning bottleneck and prevent hallucinations, the Retriever agent integrates specialized medical knowledge from external sources. It analyzes the current context (captions, questions, and reasoning history) to extract key medical concepts. These concepts are then used to query a comprehensive biomedical knowledge graph, such as SPOKE, which contains millions of medical facts. The Retriever then brings back relevant structured medical knowledge (like disease-symptom associations or anatomical relationships) and converts it into natural language to ground the reasoning process. This ensures that the AI’s responses are based on reliable, up-to-date medical expertise.
Adaptive Refinement and Few-Shot Learning
AMANDA also includes an Adaptive Reasoning Refinement mechanism, managed by the Evaluator agent. This mechanism dynamically controls how many iterations of knowledge augmentation are performed. It prevents excessive refinement, which can sometimes introduce noise, and ensures that the system stops when a sufficient level of confidence is reached, balancing accuracy with computational efficiency.
For “few-shot” scenarios, where only a small number of examples are available, AMANDA incorporates a Dual-Similarity Selection Strategy. This strategy identifies the most relevant “in-context” examples by comparing both textual and visual features, helping the Reasoner to leverage similar cases for more accurate diagnoses without needing extensive fine-tuning.
Also Read:
- Enhancing Medical Object Detection Across Diverse Imaging Modalities
- Enhancing Access to UK Clinical Guidelines with AI: A RAG System for Healthcare
Impressive Results in Data-Efficient Settings
Extensive experiments across eight different Med-VQA benchmarks have shown that AMANDA significantly improves performance in both “zero-shot” (no training examples) and “few-shot” settings. For instance, with LLaVA-Med-v1.5, AMANDA achieved an average improvement of 19.36% over the direct inference baseline. Crucially, it also demonstrated a substantial reduction in medical hallucinations, with one model showing a 47.37% reduction. This highlights AMANDA’s ability to provide more reliable and accurate AI-assisted medical diagnoses, especially in environments with limited data.
The framework’s versatility is further demonstrated by its compatibility with various Med-MLLMs and even general-purpose MLLMs, as well as different open-source and closed-source large language models as reasoning engines. This makes AMANDA a robust and adaptable solution for enhancing medical visual question answering.
