TLDR: A new study introduces AI models to automatically detect inappropriate language (IUL) in medical school curricula, which includes outdated, exclusionary, or non-patient-centered terms. Researchers found that fine-tuned small language models (SLMs) significantly outperform large language models (LLMs) like Llama-3 and GPT-4o for this task. The study also highlights that training with diverse “hard negative” examples greatly improves the accuracy of these AI systems, making specific binary classifiers more effective than multilabel approaches in many cases for mitigating harmful language in medical education.
Language plays a crucial role in shaping perceptions and behaviors, especially in medical education. The terms and expressions used in medical school curricula can profoundly influence how future healthcare professionals interact with patients, potentially contributing to health disparities. Recognizing this, a recent research paper titled “AI-Powered Detection of Inappropriate Language in Medical School Curricula” explores an innovative approach to identify and mitigate such problematic language.
Authored by Chiman Salavati, Shannon Song, Scott A. Hale, Roberto E. Montenegro, Shiri Dori-Hacohen, and Fabricio Murai from institutions like the University of Connecticut and Worcester Polytechnic Institute, this study highlights the pervasive issue of inappropriate language (IUL) in medical instructional materials. IUL encompasses outdated, exclusionary, or non-patient-centered terms that, despite the reputable origins of many materials developed over decades, are now considered inappropriate by current medical standards. Examples include using “substance abuser” instead of “having a substance use disorder” or referring to individuals as “diabetics” instead of “patients with diabetes.”
The sheer volume of medical curricular content makes manual identification of IUL prohibitively costly and impractical. To address this, the researchers conducted a first-of-its-kind evaluation of artificial intelligence models, specifically small language models (SLMs) and pre-trained large language models (LLMs), fine-tuned on a substantial dataset. This dataset comprised approximately 500 documents and over 12,000 pages, collected from the University of Washington School of Medicine.
Understanding Inappropriate Use of Language (IUL)
The study defines IUL as terms or expressions describing social identities that are inadequate by current medical standards. It focuses on the form of expression rather than the factual accuracy or intent. Key subcategories of IUL identified include:
- Gender Misuse: Using gendered terms (e.g., “women,” “men”) where anatomical or sex-based references are more appropriate.
- Sex Misuse: Incorrectly using sex terms (e.g., “male,” “female”) when referring to individuals rather than biological characteristics.
- Age Language Misuse: Vague or stigmatizing age references like “young people” or “the elderly.”
- Exclusive Language: Assuming binary sex or gender categories (e.g., “both males and females”).
- Non-Patient-Centered Language: Describing individuals primarily by their conditions (e.g., “diabetics,” “alcoholics”).
- Outdated Terminology: Terms no longer appropriate in modern medical contexts (e.g., “mentally retarded,” “fat and fertile female”).
The AI-Powered Detection System
The researchers developed a novel AI framework for IUL detection, building on the Bias Reduction in Curricular Content (BRICC) dataset. They implemented and evaluated several language models, including:
- A general binary classifier to detect the overall presence of IUL.
- Subcategory-specific binary classifiers for each IUL type.
- A multilabel classifier to identify multiple IUL subcategories simultaneously.
- A two-stage hierarchical classification pipeline, combining general detection with subcategory identification.
- Experiments with LLMs like Llama-3 and GPT-4o using few-shot prompting.
A crucial aspect of their methodology involved creating robust negative sample sets. Beyond explicitly annotated non-IUL examples (Annotated Negatives or AN), they extracted “hard negative” examples (Extracted Negatives or EN) from the larger corpus. These EN samples contained social identifiers but were deemed appropriate, challenging the models to differentiate subtle nuances between bias and true IUL.
Key Findings and Performance
The study yielded several significant insights into the performance of these AI models:
- SLMs Outperform LLMs: Both Llama-3 8B and 70B, even with carefully curated examples, were largely outperformed by the fine-tuned small language models (SLMs). While LLMs achieved high recall (identifying most IUL instances), their precision was low, leading to many false positives.
- Impact of Negative Examples: Supplementing training data with Extracted Negatives (EN) significantly boosted the Area Under the Curve (AUC) for specific classifiers by up to 25%. This indicates that providing challenging non-IUL examples helps models learn to discriminate more effectively.
- Multilabel vs. Specific Classifiers: The multilabel classifier performed best on the initial expert-annotated data. However, when the training data was augmented with the additional negative samples (AN+EN), individual subcategory classifiers often surpassed the multilabel approach, particularly for categories like Exclusive Language, Non-patient-centered Language, and Outdated Terminology. This suggests that specific classifiers offer greater robustness in diverse, real-world contexts when more negative examples are available.
This research represents a significant step towards creating automated, reliable tools to assist in the comprehensive detection of IUL patterns in medical texts. The full research paper can be accessed here.
Also Read:
- Unmasking AI Deception: Internal Probes Reveal Language Models’ Hidden Lies
- Advancing Medical AI: A Survey of Reasoning Capabilities in Large Language Models
Towards More Equitable Medical Education
The overarching goal of this work is to support human experts in systematically reviewing vast collections of medical educational texts, accelerating IUL detection while upholding quality, inclusivity, and equity. By prioritizing high recall, the system ensures that potentially harmful language is consistently flagged for human review, aligning with an “expert-in-the-loop” framework. This tool is intended as a constructive aid for improvement, fostering greater awareness and sensitivity in curriculum design, rather than a means to punish educators.
