TLDR: Foundation models, despite their success in general AI, are struggling in computational pathology. A new paper identifies seven key reasons for their underperformance, including the inherent biological complexity of human tissue, ineffective self-supervised learning for pathology images, overgeneralization, overly complex architectures, a lack of domain-specific innovation, insufficient data, and a critical mismatch in image patch sizes. The research highlights empirical evidence of low accuracy, poor robustness, and safety vulnerabilities, calling for a fundamental re-evaluation of how these AI models are designed and applied in medical contexts.
Foundation models (FMs) have brought about significant advancements in areas like computer vision and language processing, largely due to their ability to learn from vast amounts of data without explicit labels. This success naturally led to high expectations for their application in computational pathology, particularly for tasks such as cancer diagnosis, predicting disease outcomes, and retrieving multimodal information. However, a recent paper titled Why Foundation Models in Pathology Are Failing by Hamid R. Tizhoosh from Kimia Lab, Mayo Clinic, argues that these models are fundamentally struggling in this specialized medical domain.
The paper highlights several critical weaknesses observed in pathology foundation models, including low diagnostic accuracy, a lack of robustness, instability when images are geometrically altered, heavy computational demands, and concerning safety vulnerabilities. These issues, the author suggests, stem from a core conceptual mismatch between the assumptions underlying general AI foundation modeling and the inherent complexity of human tissue.
Empirical Evidence of Weaknesses
Several studies provide concrete evidence of these shortcomings. For instance, evaluations of leading pathology FMs like UNI, GigaPath, and Virchow on a large dataset of whole-slide images showed only modest performance, with significant variability across different organs. This suggests that current models might be capturing superficial texture patterns rather than true diagnostic morphology.
Another critical issue is the lack of robustness. Research indicates that many pathology FMs tend to group images based on the medical center or scanner used, rather than by the actual biological class. This ‘site bias’ leads to substantial performance drops when models are applied to data from unseen institutions, making them unreliable for widespread clinical use.
Geometric fragility is also a concern. Studies have shown that the latent representations (internal understandings) of these models can be unstable when image patches are rotated. This is particularly problematic because transformer-based architectures, common in FMs, don’t inherently account for rotation, necessitating explicit augmentation during training to achieve acceptable invariance.
Furthermore, foundation models often come with a heavy resource burden. While they can be useful in situations with limited labeled data, when ample data is available, simpler, task-specific models can match or even outperform FMs. Critically, FMs can consume significantly more energy, raising questions about their sustainability and practical superiority in clinical settings.
A surprising finding is that fine-tuning, a common method to adapt FMs to new tasks, often doesn’t work well in pathology. Instead, researchers frequently resort to ‘linear probing,’ where a simple classifier is trained on the model’s frozen features. This limitation contradicts the foundational promise of FMs to be easily adaptable and highlights that in pathology, they often function more as static feature extractors.
Finally, security and safety vulnerabilities are a serious concern. Imperceptible noise patterns, known as Universal and Transferable Adversarial Perturbations (UTAP), can collapse FM embeddings. These aren’t just malicious attacks; they also serve as diagnostic stress tests, revealing how sensitive models are to minor, real-world variations that occur naturally in the imaging process, such as differences in staining or scanner optics.
Why Pathology FMs Are Failing: A Conceptual Analysis
The paper identifies seven interrelated causes for these failures:
-
Underestimating Biological Complexity: AI often underestimates the intricate semantic complexity of human tissue. Unlike recognizing common objects, interpreting tissue morphology requires years of specialized human training, depending heavily on context, scale, and clinical correlation.
-
Ineffective Self-Supervision: Many self-supervised learning frameworks were designed for datasets with distinct, single objects. This assumption fails in histopathology, where images contain multiple, heterogeneous tissue structures, leading models to learn superficial textures rather than meaningful biological patterns.
-
The Myth of the Universal Model: The expectation that one foundation model can generalize across all organs and cancer types ignores the vast heterogeneity of pathology. Performance varies widely, underscoring the limits of a ‘one-size-fits-all’ approach.
-
Architectural Excess: Modern FMs often prioritize increasing depth and parameters without clear performance benefits. The paper suggests that progress might instead come from leaner, domain-structured architectures that better reflect the hierarchical organization of biological tissues.
-
Lack of Domain-Specific Innovation: Many pathology FMs are direct adaptations of general-purpose AI frameworks. They often lack pathology-specific mechanisms like magnification awareness, stain-invariant representations, or morphology-aware training tasks, leading to methodological stagnation.
-
Data Deficit: Pathology datasets, even the largest multi-institutional archives, are orders of magnitude smaller than the massive corpora used to train general FMs like CLIP. Privacy constraints further limit data availability, capping the value of large-scale pretraining.
-
Patch Size Mismatch: A critical issue is the discrepancy between the small image patches (e.g., 224×224 pixels) typically used by AI models and the much larger field of view required for diagnostic interpretation in microscopy. This design choice, inherited from general computer vision, means models often capture fine-grained textures but miss crucial mesoscale tissue architecture and global context.
Ethical Considerations
The rapid adoption of foundation models in pathology also raises significant ethical concerns. These include insufficient peer review and publication bias in journals, where negative results or limitations of FMs might be underreported. Researchers themselves face ethical responsibilities regarding transparent reporting, avoiding undisclosed training ‘tricks,’ and ensuring rigorous evaluation to prevent data leakage or misleading performance claims.
Perhaps most critically, there’s a risk of premature adoption by healthcare managers and administrators. Driven by institutional pressure or a desire to appear technologically advanced, these models might be implemented before their performance, generalizability, or bias have been adequately validated. Such overzealous adoption poses real dangers, including misdiagnoses and inequitable treatment, eroding public trust in medical AI.
Also Read:
- HistoLens: A New Tool for Trustworthy AI in Histopathology
- Building a Foundation for Generative AI in Healthcare: The SAGE-Health Approach
Conclusion
In essence, the paper concludes that while foundation models have revolutionized other fields, human tissue is fundamentally different from the internet data they were designed for. The current challenges faced by pathology FMs reveal a deep conceptual misalignment. True progress in computational pathology, the author argues, will require a fundamental rethinking of the paradigm itself, developing models that interpret tissue with the multi-scale, contextual, and biologically grounded understanding that pathologists possess, all within rigorous ethical and clinical frameworks.
