TLDR: Surgical-MambaLLM is a new AI model that combines Mamba2 with Large Language Models to improve Visual Question Localized-Answering (VQLA) in robotic surgery. It addresses challenges in understanding complex visual and textual details and spatial information by introducing a Cross-modal Bidirectional Mamba2 Integration (CBMI) module for fusion and a Surgical Instrument Perception (SIP) scanning mode for better spatial awareness. Experiments show it outperforms existing methods on surgical VQLA datasets.
In the evolving landscape of medical technology, intelligent robotic surgery systems are becoming increasingly sophisticated. A key area of development is Visual Question Localized-Answering (Surgical-VQLA), a task where an AI system can answer questions about surgical scenes, such as identifying organs or the location of instruments, and simultaneously highlight these areas with bounding boxes. This capability is invaluable for training medical students and junior doctors, and for enhancing the interpretative abilities of robotic systems.
However, existing methods often struggle with two main challenges: establishing complex connections between text and visual details, and accurately perceiving spatial information within the intricate environment of surgical scenes. Traditional approaches, often relying on Transformer-based models, tend to focus on global features, sometimes overlooking crucial local details necessary for precise answers, especially regarding instrument states. The complexity of laparoscopic environments also poses a significant hurdle for understanding spatial relationships.
To address these limitations, researchers have introduced a novel approach called Surgical-MambaLLM. This groundbreaking method is the first to integrate the Mamba2 model with Large Language Models (LLMs) specifically for the surgical domain. Mamba2 is known for its efficiency in capturing complex dependencies within sequences and preserving detailed information, making it well-suited to tackle the challenges of Surgical-VQLA. The core idea is to leverage Mamba2’s strengths to better understand surgical images and the relationships between visual elements and textual questions.
Surgical-MambaLLM introduces two key innovations. First, the Cross-modal Bidirectional Mamba2 Integration (CBMI) module is designed for effective multimodal fusion. Unlike previous methods that simply combine visual and textual features, CBMI uses Mamba2 to perform bidirectional scanning of both visual and textual data, creating a more robust fusion that captures intricate dependencies. This module is crucial for establishing the complex links between visual details and the questions being asked.
Second, tailored to the unique geometric characteristics of surgical scenes, the Surgical Instrument Perception (SIP) scanning mode was developed. In robotic surgery, instruments typically move from the periphery of the image towards a central target organ. Recognizing this radial pattern, the SIP scanning mode allows Mamba2 to scan surgical images from the center outwards in four directions. This specialized scanning helps the model to capture continuous regions of instrument features, maintaining their integrity and significantly enhancing Mamba2’s spatial awareness of the surgical scene. This is a crucial improvement over standard unidirectional or general bidirectional scanning methods.
The training of Surgical-MambaLLM involves a two-stage strategy. Initially, the vision encoder, CBMI module, and projector are trained while the LLM parameters are frozen. This stage ensures that the visual processing and fusion components are highly effective. In the second stage, the LLM is fine-tuned using LoRA techniques, optimizing its performance for Surgical-VQLA tasks while preserving the initial learning. The model utilizes a CLIP-ViT-B/32 pre-trained model as its vision encoder and InternLM-7B as its LLM backbone.
Extensive experiments were conducted on the EndoVis17-VQLA and EndoVis18-VQLA datasets, which are standard benchmarks for this task. The results demonstrate that Surgical-MambaLLM consistently outperforms state-of-the-art methods across various metrics, including Accuracy, F-Score, and mIoU. This superior performance highlights the model’s enhanced visual understanding, reasoning, and localization capabilities in complex surgical environments. While there is always room for further refinement, particularly in localization prediction on external validation datasets, the overall improvement is significant.
Also Read:
- Bridging Text and Vision: A New Framework for Robot Affordance Learning
- Automated Assessment for Nursing Competency Using Video-Language Models
In conclusion, Surgical-MambaLLM represents a significant leap forward in Visual Question Localized-Answering for robotic surgery. By uniquely integrating Mamba2 with LLMs and introducing specialized modules like CBMI and the SIP scanning mode, the model effectively addresses the challenges of capturing cross-modal dependencies and perceiving spatial information in surgical scenes. This innovation promises to further assist medical professionals and advance the capabilities of intelligent surgical robotic systems. You can read the full research paper here.
