TLDR: A study with 259 clinicians found that while selective AI prediction (hiding unreliable AI outputs) can restore overall diagnostic accuracy, it surprisingly leads to an increase in missed diagnoses and treatments when the AI abstains. This suggests that clinicians interpret AI deferral differently than having no AI input at all, highlighting critical trade-offs in deploying AI systems in high-stakes environments like healthcare.
Artificial intelligence (AI) holds immense promise for enhancing human decision-making across various fields, including critical areas like healthcare. However, even highly advanced AI models can sometimes produce inaccurate predictions. When humans overly rely on these incorrect AI outputs, a phenomenon known as automation bias, it can lead to worse decisions, especially in high-stakes environments like clinical diagnosis and treatment.
To address this, a strategy called selective prediction has been proposed. This approach involves the AI system withholding or hiding predictions it deems potentially unreliable, informing the user that it has abstained. The underlying assumption is that when AI defers, humans will make decisions exactly as they would without any AI involvement. But is this assumption truly valid?
A recent study, detailed in the research paper On the Limits of Selective AI Prediction: A Case Study in Clinical Decision Making by Sarah Jabbour, David Fouhey, Nikola Banovic, Stephanie D. Shepard, Ella Kazerooni, Michael W. Sjoding, and Jenna Wiens, set out to empirically test this critical assumption within a clinical context. The researchers conducted a user study involving 259 clinicians, tasking them with diagnosing and treating hospitalized patients suffering from acute respiratory failure.
The study compared clinician performance under three scenarios: without any AI assistance (baseline), with full AI input (even inaccurate predictions), and with selective prediction (where unreliable AI predictions were hidden). The findings revealed a complex picture.
Firstly, the study confirmed that inaccurate AI predictions indeed harm clinician accuracy. When clinicians were shown incorrect AI predictions, their diagnostic accuracy significantly declined compared to their baseline performance without AI. This negative impact was more pronounced among Advanced Practice Practitioners (APPs) compared to physicians, suggesting that different levels of medical training might influence reliance on AI.
Secondly, selective prediction proved effective in mitigating the negative effects of inaccurate AI. When unreliable predictions were hidden, clinician accuracy recovered to levels close to their baseline performance. This suggests that selective prediction can indeed prevent the propagation of AI errors and reduce overreliance on faulty advice.
Also Read:
- Unpacking User Engagement with AI Explanations: A Surprising Look at Trust and Decision-Making
- Unmasking Vulnerabilities: Adversarial Attacks Threaten AI Medical Questionnaire Systems
The Unexpected Trade-off: Increased Missed Diagnoses
However, the study uncovered a crucial and unexpected trade-off. While selective prediction nearly maintained overall accuracy, it significantly altered the patterns of mistakes clinicians made. When the AI abstained and informed the clinicians of its deferral, there was a notable increase in false negatives. This means clinicians were more likely to miss diagnoses and undertreat patients compared to situations where they received no AI input at all. Conversely, false positives (overdiagnosis and unnecessary treatments) decreased, which is a positive outcome.
Several factors might explain this behavior. Clinicians might interpret AI deferral not as a neutral statement, but as an implicit signal that a diagnosis is absent. Another possibility is availability bias, where clinicians might focus more on the conditions for which the AI *did* provide a prediction, making those diagnoses seem more likely. Additionally, the study found that cases where the AI deferred were perceived as more difficult by clinicians, potentially leading them to engage in a slower, more reflective (Type 2) thinking process, which research suggests can sometimes lead to more diagnostic errors.
The implications of these findings are profound. They underscore the critical need to empirically validate assumptions about how humans interact with AI within human-AI systems. Simply hiding unreliable predictions isn’t a complete solution; it introduces new challenges and changes the nature of errors. For high-risk settings like healthcare, understanding the trade-offs between false positives and false negatives is paramount. Depending on the specific clinical context, one type of error might be more tolerable than another.
This research highlights that AI systems should not be developed in isolation, optimized solely for retrospective performance. Their integration into human workflows is equally, if not more, important. Future work should focus on improving clinicians’ understanding of AI deferral and exploring how formal training might mitigate these observed error trade-offs, ensuring that AI interventions truly promote appropriate reliance across all user types.
