Systematically Revealing Implicit Biases in Medical Large Language Models

TLDR: A new framework combines knowledge graphs and auxiliary large language models to systematically identify complex, hidden biases in medical LLMs. It generates “perturbed” clinical questions by altering patient attributes and uses a multi-hop reasoning process to analyze LLM responses, proving more effective at revealing subtle biases than previous methods. This approach helps uncover how factors like age, gender, and location can influence AI-generated medical advice.

Large Language Models (LLMs) are rapidly being integrated into various applications, including critical fields like healthcare. These AI tools hold immense promise for clinical decision-making and improving patient outcomes. However, a significant concern looms over their widespread adoption: the potential for biased and unfair patterns in their operations. Identifying these biases is not just important, it’s crucial before LLMs can be reliably used in clinical settings.

A recent research paper, titled “Tell Me You’re Biased Without Telling Me You’re Biased! – Toward Revealing Implicit Biases In Medical LLMs” by Farzana Islam and Adiba Rahmatollah Beheshti from the University of Delaware, introduces a novel framework designed to systematically uncover these complex bias patterns in medical LLMs. The study highlights that while many efforts focus on evaluating bias, a rigorous and systematic framework capable of investigating nuanced biases at scale has been missing.

The Challenge of Bias in Medical LLMs

Biased behaviors in medical LLMs (Med LLMs) can have serious negative impacts on patient health equity and fairness. Previous studies have already shown risks of biased outcomes across various medical tasks, often linked to demographic factors like gender, race, or location. These biases can stem from the data used to train the models, the model design itself, or the inference procedures. Evaluating LLMs, especially when they generate free-text outputs with no single ‘right’ answer, is inherently challenging. Traditional methods, including using other LLMs as ‘judges,’ often fall short in capturing subtle or context-dependent biases, particularly those that are implicit or arise from the combination of multiple attributes (intersectional biases).

A Novel Framework for Bias Detection

The proposed framework tackles this challenge by combining Knowledge Graphs (KGs) with auxiliary LLMs. KGs provide a structured way to represent entities and their relationships, offering clarity that black-box LLMs often lack. LLMs, on the other hand, excel at contextual reasoning but can be prone to inconsistencies. By integrating these two, the framework leverages the strengths of both.

The framework operates in two main stages:

1. Perturbed Question Generation: First, a Knowledge Graph is built from clinical text or questions, extracting relationships between patient attributes like symptoms, demographics, and travel history. An ‘Attacker LLM’ then uses this KG to generate ‘perturbed questions.’ These are customized questions where specific attributes (e.g., age, gender, location) are systematically modified while keeping other information constant. This process helps to create diverse scenarios that can trigger hidden biases.
2. Answer Generation with Multi-hop Reasoning: The perturbed questions are then fed to a ‘Target LLM’ which generates answers using a three-step multi-hop reasoning process. This process involves: (a) extracting initial relationships from the question, (b) expanding these relationships by linking them to the Target LLM’s internal knowledge (like disease ontologies or demographic associations), and (c) inferring the final answer by analyzing the supporting information from the previous steps. This multi-hop approach is designed to reveal biases that might remain hidden with simpler prompting strategies.

Finally, a ‘Judge LLM’ evaluates the answers for bias, assessing how factors like age, gender, and location influence the clinical outcomes suggested by the Target LLM. This allows for scalable evaluation of bias patterns.

Experimental Validation and Key Findings

The researchers conducted comprehensive experiments using three open-source datasets (EquityMedQA, DiversityMedQA, and Nurse Bias) and various LLMs (GPT-4o, GPT-3.5-turbo, Mistral-7B, LLaMA-3.1-8B-Instruct, GPT-4.1, LLaMA-3.2-3B-Instruct). The experiments aimed to answer several key questions:

• Quality of Perturbed Questions: The perturbed questions generated by the framework were evaluated for factual consistency, clinical relevance, and coherence by multiple Judge LLMs. The results showed high validity across these criteria.
• Validity of Generated Answers: For datasets with ground truth (like Nurse Bias), the answers generated through the multi-hop reasoning process were compared against the original diagnoses. While semantic similarity slightly decreased as more attributes were perturbed simultaneously, the answers largely remained valid and meaningful.
• Effectiveness in Revealing Biases: The core finding was that the proposed framework, especially with multi-hop reasoning, consistently revealed higher bias scores compared to original questions or perturbed questions without multi-hop reasoning. This indicates its superior capability in exposing hidden bias patterns across various LLMs and demographic dimensions (age, gender, location, and their combinations).
• Human Evaluation: A survey with post-graduate participants familiar with generative AI confirmed these findings. In most clinical scenarios, participants identified the answers generated by the multi-hop method as revealing more biases, with statistically significant differences in several cases.
• Impact of Perturbation Types: An ablation study showed that perturbing multiple attributes simultaneously (e.g., age + gender + location) resulted in the highest bias scores, suggesting that compound demographic shifts amplify LLM sensitivity to social context and exacerbate disparities.

Also Read:

• Beyond the Buzz: Understanding Large Language Models in Medicine
• Bridging Logic and Language: A New Approach to Knowledge Graph Completion with LLMs

Looking Ahead

This research presents a significant step forward in understanding and identifying implicit biases in medical LLMs. By combining the structured knowledge of KGs with the reasoning capabilities of LLMs, and employing systematic perturbation and multi-hop reasoning, the framework offers a robust tool for bias evaluation. While the current study focused on age, gender, and location, future work could explore other patient factors and integrate bias mitigation strategies alongside identification.

For more detailed information, you can access the full research paper here.