TLDR: This paper introduces a framework to move from “explainable” to “explained” AI in medical imaging. It defines what constitutes a good explanation for deep learning models and proposes two key methods: (1) intuitive human-in-the-loop “sliding-window experiments” to qualitatively test and falsify explanation components, and (2) an automated quantitative approach using Vision-Language Models (VLMs) like GPT-4o to compare the predictiveness of competing explanations. The framework helps researchers rigorously test, refine, and quantify how well an explanation describes a model’s behavior, aiming to prevent models from relying on spurious features and to foster better clinical integration.
Deep learning models are increasingly used in medical image analysis, especially in areas like computational pathology. While these models often achieve high performance, their “black-box” nature means we don’t always understand how they arrive at their predictions. This lack of transparency can be problematic, as models might rely on “shortcut learning,” using irrelevant features to make predictions. For instance, a model might predict COVID-19 based on technical artifacts in a CT scan rather than actual disease indicators, or predict cancer survival based on the medical center where the data originated, not the patient’s biology. Such issues highlight that high performance metrics alone don’t guarantee a model has learned the true underlying biology.
The field of Explainable AI (XAI) aims to shed light on these black-box models. However, many XAI techniques, like saliency maps (e.g., GradCAM), are merely measurement tools that show influential features, not a full “explanation” of why a model behaves a certain way. The authors of the research paper, “From Explainable to Explained AI: Ideas for Falsifying and Quantifying Explanations”, propose a crucial shift: moving from merely “explainable” to truly “explained” AI. They define an explanation as a hypothesized natural language description of what a deep learning model does, which should be predictive of the model’s output for relevant samples.
What Makes a Good Explanation?
According to the paper, a good explanation must meet three criteria:
- Criticizable: It should be possible to experimentally test the explanation and potentially prove it wrong. For example, if a model is thought to distinguish cows from camels, you should be able to test it with images of cows in deserts or camels in meadows to see if it still performs as expected.
- Hard to Vary: The explanation should be specific and not easily changed without losing its predictive power.
- Non-Authoritarian: The validity of the explanation should not depend on who proposed it. It should stand on its own merits, based on evidence.
Testing Explanations with Sliding-Window Experiments
The researchers introduce a human-in-the-loop system to test and potentially falsify components of an explanation. This involves “sliding-window experiments” using an AI-integrated slide viewer, particularly useful for whole-slide images in computational pathology. The idea is to observe how the model’s prediction changes as specific features are included or excluded from the field of view. This allows researchers to intuitively find and select patches that challenge the initial explanation.
For example, when testing a colorectal cancer risk prediction model, the initial explanation suggested that well-differentiated tissue leads to a lower risk score. However, sliding-window experiments sometimes showed that healthy glandular tissue actually received high risk scores, contradicting the explanation. Similarly, while the explanation suggested immune response decreases risk, the experiments found cases where lymphocytic infiltration increased the risk score. These observations highlight that even seemingly good explanations might be incomplete or inaccurate, and the tool helps refine them.
Quantifying Explanations with Vision-Language Models
Beyond qualitative testing, the paper proposes a method to quantitatively compare competing explanations using a general-purpose Vision-Language Model (VLM), specifically GPT-4o. The process involves providing the VLM with an image description and a hypothesized explanation, then prompting it to predict the black-box model’s output (e.g., high, medium-high, medium-low, low risk). The VLM’s predictions are then compared to the actual model’s output using metrics like AUROC (Area Under the Receiver Operating Characteristic curve).
The experiments showed that “poor” explanations (like a “cow-camel” explanation for pathology images, or an intentionally “inverse” explanation) resulted in random or worse-than-random performance, as expected. Conversely, “predictive” explanations, such as a coarse “tumor-lymphocyte” explanation or a more “detailed analysis” explanation, achieved significantly higher AUROC scores, especially on samples with very clear high or low risk predictions. This demonstrates that the framework can distinguish between effective and ineffective explanations, allowing researchers to systematically quantify the predictive power of a refined explanation on a larger dataset without requiring extensive human annotations.
Also Read:
- MedReasoner: Advancing Medical Image Analysis with AI Reasoning and Precision Grounding
- Navigating the Pitfalls: Hallucinations in AI-Powered Medical Imaging
Towards Better AI in Medicine
The authors emphasize that rigorously testing and iterating on explanations is crucial for the clinical integration of deep learning models in medical image analysis. By providing conceptual frameworks and practical tools for falsification and quantification, this work offers a path to better understand AI models, reduce the likelihood of unwanted behavior, and potentially uncover novel biological insights. Future work might involve multi-agent systems that automatically generate, challenge, and refine explanations, leading to more robust and trustworthy AI in healthcare.
