Unpacking LLM Performance in Healthcare: The Critical Role of Diverse Evaluation

TLDR: A new research paper introduces an infrastructure to rigorously test medical LLMs for biases, hallucinations, and omissions across diverse patient scenarios. The study found that different LLM evaluators exhibit low agreement (average Cohen’s Kappa of 0.118) and that statistically significant results from one LLM evaluation setup do not guarantee generalizable conclusions. The authors recommend using multiple LLMs as evaluators and publishing inter-LLM agreement metrics to ensure more robust and transparent evaluations of AI in healthcare.

Large Language Models (LLMs) are rapidly transforming various sectors, and healthcare is no exception. These advanced AI systems hold immense promise for patient-facing medical applications, offering the ability to communicate complex medical information in natural, understandable language. This can lead to improved patient engagement and satisfaction, and potentially help bridge healthcare gaps across different demographic and geographic groups.

However, the integration of LLMs into sensitive medical contexts comes with significant concerns. Issues like hallucinations (generating false information), omissions (missing crucial details), and biases towards specific demographic groups are well-documented. In healthcare, such errors can lead to patient harm, dissatisfaction, and a critical loss of trust in medical professionals and technology.

Investigating LLM Failures in Patient Scenarios

A recent research paper, titled “Demo: Statistically Significant Results On Biases and Errors of LLMs Do Not Guarantee Generalizable Results,” delves into the conditions under which LLMs deviate from expected behavior in patient-facing scenarios. The study, conducted by Jonathan Liu, Haoling Qiu, Jonathan Lasko, Damianos Karakos, Mahsa Yarmohammadi, and Mark Dredze, explores how different LLMs exhibit hallucinations, omissions, and biases when presented with hundreds of simulated mental health patient scenarios.

The researchers developed a novel infrastructure designed to rigorously probe LLMs. This system has two main components:

1. Automatic Query Generation: A sophisticated pipeline that creates realistic patient questions by sampling a wide range of patient demographics (age, gender, race), medical histories, disorders, and even writing styles. This ensures that the LLMs are tested against a diverse and representative set of real-world queries.

2. Comprehensive Answer Evaluation: An evaluation pipeline that uses multiple LLM-as-a-judge setups and prompts. This includes advanced hallucination and omission detection, as well as agentic workflows where critic agents provide feedback to refine detections. The system also employs LLM-as-a-judge to categorize treatment suggestions (e.g., manage symptoms at home, visit a doctor, seek resources).

Key Findings: Low Agreement and Non-Generalizable Results

The study’s baseline experiments revealed several critical insights into the reliability of LLM evaluations:

• Low Inter-LLM Agreement: When different LLMs were used as evaluators, they exhibited surprisingly low agreement scores. The average Cohen’s Kappa, a statistical measure of inter-rater agreement, was a mere 0.118. For context, a Kappa of 0.60 is often considered a standard for strong agreement among human annotators. This suggests that different LLM evaluators can interpret the same answers very differently.

• Impact of LLM Pairs: The choice of both the answering LLM and the evaluation LLM significantly affected the downstream evaluation tasks. The study found that only specific pairs of answering and evaluation LLMs yielded statistically significant differences across various factors like writing styles, genders, and races. This means that conclusions drawn from one LLM pair might not hold true for another.

• Conflicting Conclusions: In some cases, using data from different evaluation LLMs led to opposite, yet statistically significant, conclusions. For example, one evaluator might show a higher “Visit” rate for a certain writing style, while another evaluator for the same responses might show a lower rate for that same style.

These findings highlight a crucial challenge: statistically significant results from a single LLM evaluation setup do not guarantee generalizable or reliable conclusions, especially in the absence of ground-truth data.

Also Read:

• Assessing AI’s Reliability and Hallucination Risks in Spine Surgery Decisions
• Evaluating AI’s Precision in Clinical Text: Introducing the MEDRECT Benchmark

Recommendations for Robust LLM Evaluation

Based on their extensive analysis, the researchers offer two key recommendations for future studies involving LLM evaluation, particularly in critical domains like healthcare:

1. Use Multiple LLMs as Evaluators: To mitigate the risk of obtaining statistically significant but non-generalizable results, studies should employ several different LLMs as evaluators. This approach helps to provide a more comprehensive and balanced assessment.

2. Publish Inter-LLM Agreement Metrics: For transparency and to allow authors and readers to better understand the nature of the questions and the behavior of the LLM evaluators, it is considered best practice to report inter-LLM agreement metrics, similar to how human-annotator agreement is reported.

The researchers have made their code and a substantial dataset (3.2 million prompts, 29 thousand answers, and 684 thousand evaluations) publicly available, which is a significant contribution to the field. This dataset is unique in measuring the effect of varying both the response and evaluator LLM in the medical domain. You can find more details about their work and access the resources at the research paper link.

This research underscores the complexity of evaluating AI in healthcare and provides a valuable framework for developing more robust and trustworthy medical chatbots.