TLDR: The paper investigates “model multiplicity” in medical AI, where equally good models can make different predictions for the same patient. It shows that relying on a single model leads to arbitrary diagnoses and that standard validation metrics are unreliable. The study proposes using small ensembles of models with an abstention strategy to significantly reduce arbitrary predictions, improving diagnostic reliability. It also finds that higher model accuracy, often achieved through increased model capacity, helps reduce predictive multiplicity.
In the rapidly advancing field of artificial intelligence, particularly within healthcare, the reliability of machine learning models is paramount. A critical, yet often overlooked, challenge is ‘model multiplicity’ – a phenomenon where multiple AI models can perform equally well on average, but produce different predictions for the same individual case. This issue, also known as the Rashomon Effect, poses significant risks in medicine, where a patient’s diagnosis and treatment could inadvertently depend on arbitrary choices made during model development.
A recent study, titled On Arbitrary Predictions from Equally Valid Models, delves deep into this problem. Conducted by a team including Sarah Lockfisch, Kristian Schwethelm, Martin Menten, Rickmer Braren, Daniel Rückert, Alexander Ziller, and Georgios Kaissis, the research empirically analyzes the extent, causes, and consequences of predictive multiplicity across various medical tasks and model architectures.
The study highlights a crucial flaw in standard AI development: validation metrics, which are typically used to select the ‘best’ model, often fail to identify a truly unique or optimal model. This means that a substantial portion of AI predictions can hinge on seemingly arbitrary decisions, such as the initial random settings used when training a model. For instance, if a single model is deployed in a clinical setting, a patient’s diagnosis might be influenced more by the randomness of the training process than by the actual characteristics of their data.
To investigate this, the researchers trained 1,400 models across five diverse medical imaging modalities (abdominal CT, blood cell images, breast ultrasound, OCT scans, and X-rays) and four common deep learning architectures (ResNet50, GC ViT, EfficientNet, ConvNeXt). The only variable changed between models of the same configuration was the random seed used for initialization. Their findings were striking:
Also Read:
- Knowledge Grafting: A New Method for Efficient AI on Resource-Limited Hardware
- Navigating Hidden Information: How AI Agents Model Uncertainty in Games
Key Insights from the Research
Firstly, the study found that validation performance is an unreliable indicator of how well a model will generalize to new, unseen data. Models that appeared to perform similarly during validation often showed significant differences when tested on real-world data. This suggests that choosing a single model based solely on its validation score is an arbitrary decision among many equally valid alternatives.
Secondly, the research confirmed that individual patient predictions are indeed arbitrary when relying on a single model. Even in scenarios where models achieved very high average accuracy (e.g., over 99%), a notable percentage of samples received conflicting predictions across different, equally plausible models. This underscores the risk of deploying a single, arbitrarily chosen model, as it could lead to ambiguous or even harmful diagnoses for patients.
Thirdly, the study proposed a practical solution: using small ensembles of models combined with an abstention strategy. This approach involves leveraging multiple models and only making an automated prediction when there is a high consensus among them. If models disagree, the system abstains, deferring the decision to a human expert. The research demonstrated that even small ensembles (as few as two to five models) can effectively mitigate or eliminate measurable predictive multiplicity, significantly improving diagnostic reliability. This consensus-driven approach not only enhances consistency but also often leads to higher accuracy on the predictions that are made.
Finally, the paper explored the relationship between model capacity (the complexity of the model) and predictive multiplicity. Counter-intuitively, they found that higher model capacity, which typically leads to improved accuracy, actually reduces predictive multiplicity. This suggests that while accuracy alone isn’t a complete solution, maximizing accuracy through more capable models can help minimize arbitrary predictions.
In conclusion, this research underscores the critical importance of acknowledging and addressing model multiplicity in medical AI. It challenges the traditional paradigm of selecting a single ‘best’ model and advocates for ensemble-based strategies to build more robust, consistent, and trustworthy diagnostic systems. By deferring uncertain cases to expert review, AI can become a more reliable and safer tool in high-stakes medical decision-making.
