TLDR: This survey provides a detailed review of multimodal deep learning in ophthalmic diagnostics, covering advancements up to 2025. It categorizes methods into task-specific approaches (for detection, diagnosis, and data generation) and large-scale foundation models (including MIM-based vision models, CLIP-style models, and multimodal large language models). The paper discusses how integrating diverse imaging and clinical data enhances diagnostic accuracy and generalization for diseases like diabetic retinopathy, glaucoma, and age-related macular degeneration. It also highlights persistent challenges such as data heterogeneity, annotation scarcity, interpretability, and the need for greater population diversity in datasets, while outlining future directions like ultra-widefield imaging and reinforcement learning-driven reasoning.
Visual impairment is a significant global health issue. To accurately diagnose eye conditions, ophthalmologists often rely on a variety of imaging techniques, each providing different but complementary information. This comprehensive review explores the latest advancements in using deep learning, a powerful form of artificial intelligence, to integrate these diverse imaging sources for better eye diagnostics.
Traditionally, doctors combine information from multiple sources like color fundus photography (CFP), optical coherence tomography (OCT), and angiography to get a complete picture of retinal health. This approach offers several advantages over using a single imaging method: it enhances diagnostic accuracy, makes the diagnostic models more robust, and improves their ability to generalize across different cases.
Task-Specific Approaches in Eye Diagnostics
The survey categorizes current deep learning methods into two main groups. The first group, ‘task-specific multimodal approaches,’ involves AI models designed for particular clinical applications. These models are highly specialized and optimized for specific combinations of imaging and non-imaging data to address clearly defined ophthalmic tasks.
For instance, these models are used for:
- Detection and Segmentation: This involves identifying and outlining anatomical structures and disease-related lesions, such as drusen in age-related macular degeneration (AMD), or segmenting thin, low-contrast blood vessels and the optic disc/cup structures crucial for glaucoma diagnosis. Multimodal data significantly improves the accuracy of these tasks.
- Multimodal Diagnosis: These models integrate structural, functional, and clinical data for early detection and precise classification of eye diseases. For example, combining CFP with OCT can greatly improve the detection of early-stage AMD, which might be missed with CFP alone. Similarly, for diabetic retinopathy (DR), combining CFP with OCT angiography (OCTA) leads to more accurate identification of subtle microvascular changes.
- Data Generation and Augmentation: AI can generate new ophthalmic images or descriptions. This is particularly useful when labeled data is scarce. For example, generative models can convert standard CFP images into angiography images, allowing non-invasive assessment of lesions. They can also automatically produce clinical reports from images, enhancing efficiency and providing decision support.
The Rise of Foundation Models in Ophthalmology
The second major category is ‘large-scale multimodal foundation models.’ These are massive AI systems pre-trained on vast amounts of unlabeled and diverse data from multiple sources. This extensive training allows them to adapt effectively to a wide range of ophthalmic tasks with minimal additional fine-tuning.
These foundation models include:
- MIM-based Vision Models: These models, like RETFound and EyeFound, are pre-trained using techniques that involve reconstructing masked parts of images. They learn powerful representations from large collections of unlabeled retinal images, demonstrating strong performance in diagnosing conditions like diabetic retinopathy and glaucoma.
- CLIP-style Models: Inspired by the CLIP architecture, these models align ophthalmic images with corresponding clinical text reports using contrastive learning. This alignment improves cross-modal understanding and makes the models more interpretable, helping to bridge the gap between imaging findings and clinical decision-making. Examples include RET-CLIP and EyeCLIP.
- Multimodal Large Language Models (MLLMs): These are the most generalized models, combining visual encoders with large language models. They enable complex reasoning and generation tasks, such as diagnostic report generation, disease progression prediction, and natural language-based clinical decision support. VisionUnite and EyecareGPT are examples tailored for ophthalmology.
- Fine-tuned Foundation Models: These approaches adapt general-purpose pre-trained models for specific ophthalmic tasks. By making minimal architectural adjustments and using focused domain data, they achieve excellent performance efficiently. OphGLM and DeepDR-LLM are examples that enhance diagnostic accuracy and provide clinical decision support.
The development and evaluation of these advanced AI models heavily rely on high-quality, diverse datasets. The survey highlights various publicly available datasets specifically for AMD, DR, and glaucoma research, as well as general ophthalmic studies. These datasets vary in scale, imaging modalities covered, and accessibility, reflecting the broad scope of research in ophthalmic AI.
Also Read:
- Unraveling AI’s Multimodal Decisions: A Review of Explainability in Attention Models
- Precision in Retinal Imaging: A Novel Loss for Artery/Vein Classification
Challenges and Future Outlook
Despite these encouraging advancements, several challenges remain. Data from different devices and clinics can be highly variable, making it difficult for AI models to generalize. High-quality annotations, especially for detailed lesion segmentation, are expensive and time-consuming to obtain. Furthermore, the ‘black box’ nature of many deep learning models limits their interpretability, which is crucial for building trust among clinicians and gaining regulatory acceptance.
The ‘modality gap’ also poses a problem, as current fusion methods often fail to fully exploit the complementary advantages of different imaging types. Finally, the limited diversity in existing datasets, often skewed towards specific ethnic or socioeconomic groups, raises concerns about algorithmic bias and generalization to diverse patient populations.
Looking ahead, future research aims to address these limitations. Promising directions include the use of ultra-widefield (UWF) imaging, which captures a much broader view of the retina, allowing for earlier and more holistic disease characterization. Another exciting area is the development of reinforcement learning-based reasoning frameworks. These AI systems aim to mimic the step-by-step decision-making processes of clinicians, providing more explainable and logically coherent diagnostic paths, especially valuable in situations with limited annotated data. This will lead to more intelligent, interpretable, and clinically applicable AI systems for ophthalmology. For more in-depth information, you can refer to the full research paper here.
