TLDR: This research paper explores how the explainability of Artificial Intelligence (AI) systems in healthcare influences the distribution of liability among medical practitioners, hospitals, and AI manufacturers. It contrasts opaque “Oracle” AI with transparent “AI Colleague” AI, arguing that explainability is crucial for establishing a clear responsibility framework, shaping professional behavior, and mitigating the practice of defensive medicine. The authors conclude that explainability, alongside accuracy, should be a primary focus for AI certification in healthcare.
The integration of Artificial Intelligence (AI) into the healthcare sector promises revolutionary advancements, but it also introduces complex questions, particularly concerning liability. A recent research paper, titled “Explainability matters: The effect of liability rules on the healthcare sector,” delves into how the transparency of AI systems can profoundly influence who is held responsible when things go wrong, and how this impacts medical practice.
The paper, authored by Jiawen Wei, Elena Verona, Andrea Bertolini, and Gianmarco Mengaldo, highlights a critical distinction between two types of AI systems: the “Oracle” and the “AI Colleague.” The “Oracle” represents an opaque AI, a black box that provides diagnoses or recommendations without explaining its reasoning. In contrast, the “AI Colleague” is a fully explainable AI that can justify its conclusions in a manner comprehensible to human medical professionals.
The Liability Dilemma with Opaque AI (The “Oracle”)
When a medical practitioner uses an “Oracle” AI, they receive an output without understanding how the AI arrived at its conclusion. The paper argues that in such scenarios, practitioners might be incentivized to simply agree with the AI’s recommendations, even if they have doubts. This is because, from a legal standpoint, aligning with an AI’s output could potentially shift or share liability with the AI manufacturer if an error occurs. If the AI is wrong, the practitioner might argue that the system was defective, leading them to a mistake. This strategic behavior, driven by the fear of litigation, can lead to what is known as “defensive medicine.”
Defensive medicine involves medical professionals making decisions primarily to minimize their risk of lawsuits, rather than solely focusing on the patient’s best interest. This could mean ordering unnecessary tests or avoiding high-risk treatments. The paper suggests that opaque AI systems could exacerbate this problem, as practitioners might blindly follow AI advice to protect themselves, potentially compromising patient care.
The Role of Explainable AI (The “AI Colleague”)
The “AI Colleague” model presents a different picture. With an explainable AI, practitioners can understand the reasoning behind a diagnosis or recommendation, much like they would when consulting with a human colleague. This transparency empowers the doctor to critically assess the AI’s output, leveraging their own expertise and professional judgment. In this scenario, the practitioner retains primary responsibility for their decisions.
If a practitioner agrees with a wrong explainable AI, they could be held liable because the explanation should have allowed them to identify the error. Conversely, if they disagree with a correct explainable AI, they might also be held responsible for not recognizing the correct reasoning. This framework encourages practitioners to engage actively with the AI, using it as a sophisticated tool to support their expertise, rather than an unquestionable authority.
Also Read:
- Navigating the Future: Enhancing Trust and Safety in Autonomous Shipping Through Explainable AI
- The STAR-XAI Protocol: Building Transparent and Self-Correcting AI Agents
Shifting Responsibility and Policy Implications
The paper argues that explainability is key to establishing a clear responsibility framework. For an “Oracle” AI, liability might frequently swing between the practitioner and the manufacturer, creating ambiguity. For an “AI Colleague,” the manufacturer’s liability would be more limited, focusing on cases where the AI’s explanation itself is erroneous or misleading, rather than just a diagnostic error. This distinction is crucial for policymakers and certification bodies.
The authors emphasize that when certifying AI systems for diagnostics, the focus should not be solely on overall accuracy but also, and perhaps more importantly, on explainability. By prioritizing explainability, AI can truly serve as a valuable assistant to medical professionals, fostering a collaborative environment that enhances patient safety and mitigates the risks of defensive medicine. This research provides a compelling argument for why explainability matters not just for trust, but for the very structure of legal and ethical responsibility in AI-driven healthcare. You can read the full paper here.
