TLDR: This research paper systematically reviews foundation models (FMs) in medical image analysis, categorizing them into vision-only (VFMs) and vision-language (VLFMs). It analyzes their architectures, training strategies, and clinical applications across various modalities. The paper highlights the rapid growth of FMs, their diverse applications in segmentation, classification, and report generation, and discusses key challenges like domain adaptation, computational constraints, interpretability, and fairness, proposing future research directions for robust clinical integration.
Foundation models (FMs), a groundbreaking advancement in artificial intelligence, are rapidly transforming the landscape of medical image analysis. These powerful AI models, unlike their traditional, task-specific predecessors, are designed to learn broad, generalized representations from vast amounts of labeled and unlabeled multimodal data. This allows them to adapt to a wide array of clinical applications—from segmenting anatomical structures to generating detailed medical reports—with minimal fine-tuning.
A recent systematic review and meta-analysis titled “Foundation Models in Medical Image Analysis: A Systematic Review and Meta-Analysis” provides a comprehensive look into this evolving field. The paper, authored by Praveenbalaji Rajendran, Mojtaba Safari, Wenfeng He, Mingzhe Hu, Shansong Wang, Jun Zhou, and Xiaofeng Yang, addresses the fragmentation in current research by systematically categorizing and analyzing FMs based on their architectural foundations, training strategies, and clinical applications across different imaging modalities. For more details, you can read the full paper here.
Understanding Foundation Models in Medical Imaging
Historically, interpreting medical images like X-rays, CT scans, and MRIs relied heavily on the expertise of medical professionals. However, as imaging data became more complex, manual interpretation grew time-consuming and prone to inconsistencies. Deep learning (DL) offered a new paradigm, but conventional DL models often required extensive, annotated datasets and were limited to specific tasks, hindering their scalability.
Foundation models overcome these limitations by leveraging large-scale architectures trained on diverse, often unlabeled, datasets. This allows them to generalize across many tasks with “zero-shot” (no new examples) or “few-shot” (minimal new examples) adaptation. Originating from natural language processing (NLP) with models like BERT and GPT, FMs have successfully expanded into computer vision (Vision Transformers or ViTs) and multimodal contexts (like CLIP, which aligns visual and textual data).
In medical imaging, FMs are broadly categorized into two main types: vision-only models (VFMs) and vision-language models (VLFMs). VFMs process image data exclusively, excelling in tasks like anatomical segmentation and disease classification. VLFMs, on the other hand, integrate both visual and textual information, enabling a deeper understanding of clinical context for applications such as automated report generation and visual question answering.
Architectural Backbones and Learning Approaches
The core of FMs lies in their architectural backbones, which extract high-level semantic representations. These have evolved from traditional Convolutional Neural Networks (CNNs), known for hierarchical feature learning (e.g., U-Net for segmentation), to more flexible Transformer-based architectures. Transformers, with their self-attention mechanisms, can capture long-range dependencies in data, making them highly scalable. Many modern FMs utilize hybrid architectures, combining the strengths of both CNNs and Transformers.
Training FMs involves advanced learning paradigms, primarily self-supervised learning (SSL) and semi-supervised learning (Semi-SL). SSL allows models to learn from vast unlabeled data by generating their own supervisory signals. This includes predictive tasks (like predicting masked image regions), generative tasks (like Masked Autoencoders or Diffusion Models that reconstruct missing data or generate new images), and contrastive learning (where models learn to distinguish similar from dissimilar data pairs, as seen in CLIP).
Semi-supervised learning bridges the gap by using a small amount of labeled data alongside large unlabeled datasets, which is particularly useful in medical imaging where annotations are scarce. Techniques like consistency regularization and pseudo-labeling are common here.
Vision-Only Foundation Models (VFMs) in Detail
The Segment Anything Model (SAM) is a prominent VFM that revolutionized segmentation by generating masks from various prompts (points, bounding boxes, text) without specific fine-tuning. While powerful in natural images, direct application of SAM to medical images faced challenges due to domain differences. This led to numerous adaptations:
- Foundational Adaptations: Models like MedSAM and SAM-Med2D fine-tuned SAM on large medical datasets. SAM-Med3D extended this to 3D volumetric data, crucial for modalities like CT and MRI.
- Prompt Optimization: Instead of modifying SAM’s core weights, methods like AutoSAM and ESP-MedSAM focus on automatically generating or refining prompts to enhance segmentation performance.
- Hybrid Architectures: SAM has been integrated with other deep learning techniques, such as CNNs (SAM-UNet, MedSAM-CA) and LSTMs, to combine its generalization capabilities with efficient local feature extraction.
- Efficient and Lightweight SAM: To address computational demands, variants like De-LightSAM and LiteMedSAM compress the architecture, making them suitable for resource-constrained clinical settings.
- Domain-Specific SAMs: Adaptations have been developed for specific medical domains, including pathology (WSI-SAM, CellSAM), ultrasound imaging (SonoSAM, SAMUS), surgical and endoscopic imaging (SurgicalSAM, Polyp-SAM), and anatomy-specific tasks (PCa-SAM for prostate cancer, Mammo-SAM for breast masses).
Beyond SAM, other generalist VFMs exist for universal segmentation (UniverSeg, TotalSegmentator), disease classification (RETFound for retinal images, CheXFound for chest X-rays, Prov-GigaPath for pathology), and image registration and synthesis (uniGradICON, MedDiff-FM).
Vision-Language Foundation Models (VLFMs) in Detail
VLFMs are crucial for tasks requiring the integration of visual and textual information. Contrastive learning is a dominant paradigm here, aligning image and text embeddings in a shared space. CLIP is a foundational framework that has inspired many medical adaptations:
- Foundational and Data-efficient Adaptations: Models like GLoRIA and LIMITR learn global and local alignments between medical images and radiology reports. MedCLIP and PMC-CLIP address data scarcity by leveraging weak supervision or large-scale biomedical literature to pre-train models.
- Knowledge-Enhanced Adaptations: Frameworks such as MedKLIP and ARL explicitly incorporate external medical knowledge (e.g., from UMLS) to reduce semantic ambiguity and improve cross-modal alignment.
- Generalist Contrastive VLFMs: Models like Med-UniC and UniMed-CLIP aim for task-agnostic performance across diverse datasets, modalities, and even languages.
- Task-Specific Contrastive VLFMs: These are tailored for narrow clinical contexts, such as segmentation (MedCLIP-SAM), modality-specific diagnosis (FetalCLIP for fetal ultrasound, EchoCLIP for echocardiograms), and language-centric tasks like report generation (BioViL-T) and visual question answering (MUMC, ELIXR).
Generative VLFMs, leveraging techniques like masked representation modeling and diffusion-based synthesis, are also emerging. Examples include RadFM for radiology report generation, Uni-Med for multi-task learning, and Lingshu for enhanced clinical reasoning. Interactive VLFMs, like LLaVA-Med and CXR-LLaVA, enable dynamic, dialogue-driven interactions, allowing clinicians to query AI systems using natural language.
Also Read:
- Large AI Models Reshape Neuroscience Research and Clinical Practice
- Enhancing Clinical Decision Support with Specialized Medical Image Captioning
Trends and Future Outlook
The meta-analysis reveals a significant increase in both VFM and VLFM publications, particularly from 2021 to 2025. Public datasets, especially those for CT and X-ray, are predominantly used for training and evaluation. Segmentation remains the most common downstream task for VFMs, while VLFMs are increasingly applied to report generation, detection, localization, and reasoning tasks.
The field faces ongoing challenges, including the debate between universal (broadly applicable) and specialist (highly accurate in narrow contexts) FMs. Hybrid strategies, combining universal backbones with lightweight domain-specific adaptations, are emerging as a promising middle ground. Computational demands, efficient fine-tuning, and interpretability are also key concerns.
Crucially, the evaluation of FMs extends beyond traditional accuracy. It encompasses representation quality (cross-modal similarity, clustering, linear separability), robustness to real-world variations (cross-domain testing, long-tail evaluation), and interpretability (grounding predictions to specific regions, providing retrieval-based evidence). Fairness and equity are paramount, as FMs can inherit and amplify biases from training data. Addressing these biases requires integrated strategies throughout the model’s lifecycle, from data curation to deployment monitoring, ensuring equitable performance across diverse patient populations.
The continuous evolution of foundation models promises to accelerate their translation into real-world medical practice, enhancing diagnostic accuracy, improving clinical workflows, and ultimately contributing to better patient care.
