TLDR: EndoAgent is a novel AI system designed to improve endoscopic image diagnosis. It utilizes a unique dual-memory architecture (short-term for action tracking, long-term for experiential learning) and integrates a suite of specialized tools for tasks like lesion classification, detection, and report generation. The system performs multi-step reasoning with reflection, enabling it to iteratively refine decisions and enhance accuracy. Evaluated on a new benchmark called EndoAgentBench, it consistently outperforms existing general and medical AI models, demonstrating strong flexibility and reasoning capabilities for complex clinical workflows.
Endoscopy is a crucial procedure for diagnosing and treating conditions in the digestive tract. However, the quality of diagnosis often depends heavily on the physician’s experience. While artificial intelligence (AI) has shown promise in assisting with tasks like lesion detection, existing AI models often struggle with the complex, multi-step processes involved in real-world clinical workflows and lack the ability to adapt to new tasks.
To address these challenges, researchers have introduced a novel AI system called EndoAgent. This is the first memory-guided AI agent designed for endoscopic analysis, aiming to bridge the gap between visual information and clinical decision-making. EndoAgent stands out by integrating iterative reasoning with adaptive tool selection and collaboration, mimicking how human experts approach complex cases.
How EndoAgent Works: A Dual-Memory Approach
At the heart of EndoAgent is its unique dual-memory design. This system allows for sophisticated decision-making by maintaining logical coherence through a ‘short-term memory’ that tracks recent actions and outputs. Simultaneously, a ‘long-term memory’ progressively enhances the agent’s reasoning by storing ‘experiential learning’ in the form of reflective feedback, such as identified errors or uncertainties from previous rounds of analysis.
This memory-guided workflow enables EndoAgent to refine its decisions iteratively, much like a clinician learns from experience. It can adapt its tool selection and reasoning strategies based on accumulated knowledge, leading to improved accuracy over time.
A Suite of Expert Tools for Diverse Tasks
To support a wide range of clinical tasks, EndoAgent integrates a comprehensive suite of expert-designed tools within a unified reasoning loop. These specialized tools cover six core endoscopic tasks:
- Classification: Identifying the type of tissue or lesion (e.g., normal, polyp, adenoma, cancer).
- Detection: Accurately locating lesion areas and providing spatial information.
- Segmentation: Performing pixel-level delineation of lesions and tools.
- Visual Question Answering (VQA): Answering clinically relevant questions directly from image content.
- Image Editing: Generating or removing synthetic lesions, useful for training and data augmentation.
- Report Generation: Automatically synthesizing outputs from various modules to create standardized medical reports.
This modular and context-aware orchestration ensures that EndoAgent can flexibly tackle diverse clinical subtasks, applying the right expertise at the right moment.
EndoAgentBench: A New Standard for Evaluation
To systematically evaluate the performance of AI agents in endoscopy, the researchers also introduced EndoAgentBench. This comprehensive benchmark comprises 5,709 visual question-answer pairs, designed to assess both fine-grained visual understanding and open-ended language generation capabilities in realistic clinical scenarios. It covers five key diagnostic subtasks: lesion classification, lesion quantification, visual grounding, image captioning, and report generation.
EndoAgentBench is built upon a diverse dataset, combining widely used public endoscopic image datasets with a significant portion of private clinical data, ensuring both generalizability and clinical authenticity. This benchmark provides a robust foundation for comparing and advancing AI models in the endoscopy domain.
Also Read:
- AI Framework Uses Synthetic Data for Better Polyp Detection in Colonoscopies
- How TeamMedAgents Uses Human Teamwork to Boost Medical AI Accuracy
Performance and Scalability
Extensive experiments have shown that EndoAgent consistently outperforms both general-purpose and medical multimodal models across various tasks. For instance, it achieved superior accuracy in lesion classification, quantification, and visual grounding, demonstrating the effectiveness of its multi-round reflection framework and integrated toolset. In language generation tasks, such as image captioning and medical report generation, EndoAgent also achieved top performance, often surpassing other state-of-the-art models.
The research also highlighted the importance of the reflection and dual-memory mechanisms, showing a significant drop in performance when these components were removed. Furthermore, the study demonstrated EndoAgent’s scalability and robustness, as it maintained consistent performance even when its core large language model was swapped with alternatives like Gemini 2.5 Pro or Claude 3.7 Sonnet.
EndoAgent represents a significant step forward in developing general AI systems for endoscopic image diagnosis. Its memory-guided reflective architecture and integrated toolset offer a flexible and powerful approach to complex clinical reasoning. For more technical details, you can refer to the full research paper: EndoAgent: A Memory-Guided Reflective Agent for Intelligent Endoscopic Vision-to-Decision Reasoning.
