TLDR: A new study reveals that AI-powered medical advice systematically differs across patient social groups. Indigenous and intersex patients receive less readable and more complex guidance, with these disparities significantly amplified for intersectional identities. The research highlights an urgent need for AI developers to investigate and mitigate these biases to ensure equitable and accessible healthcare support for all.
Large Language Models (LLMs) are rapidly transforming how the public accesses health information, offering personalized answers to medical questions. These AI tools are becoming increasingly sophisticated, even outperforming human professionals in some medical areas, and hold significant promise for providing accessible, low-cost healthcare support, particularly in underserved regions.
However, a recent study titled “DR. BIAS: SOCIAL DISPARITIES IN AI-POWERED MEDICAL GUIDANCE” by Emma Kondrup and Anne Imouza, accepted at the Symposium on Model Accountability, Sustainability and Healthcare 2025, reveals a critical oversight: current evaluations often ignore the social complexities of healthcare and existing health disparities. The research highlights how biases can inadvertently translate into LLM-generated medical advice, potentially impacting users in unjust ways. You can read the full paper here: DR. BIAS: SOCIAL DISPARITIES IN AI-POWERED MEDICAL GUIDANCE.
Uncovering Systemic Differences in AI Medical Advice
The study conducted an exploratory analysis of LLM responses to a series of medical questions across key clinical domains. To understand how advice might differ, the researchers simulated these questions being asked by various patient profiles, carefully varying factors like sex, age range, and ethnicity. By comparing the natural language features of the generated responses, a concerning pattern emerged: LLMs systematically provide different advice based on a patient’s social group.
Specifically, the findings indicate that Indigenous and intersex patients receive medical advice that is notably less readable and more complex. These disparities become even more pronounced when considering intersectional groups – for example, intersex individuals who also identify with Indigenous or Black ethnic groups faced the most significant challenges in receiving clear, concise advice.
Key Findings on Readability and Complexity
The research delved into readability metrics, such as advice length, Flesch reading ease (a measure of how easy text is to read, with higher scores indicating easier text), and assessed grade level. While differences between advice for female and male patients were generally minor, intersex individuals consistently received longer, more intricate advice. Their Flesch reading ease scores were significantly lower, and their assessed grade levels nearly two points higher than those for male and female patients, indicating much harder-to-read content.
Similar trends were observed across different ethnic groups. American Indian or Alaska Native (AIAN), Native Hawaiian or Pacific Islander (NHPI), and Black or African American (BAA) patients frequently received more complex and lengthy advice. This was particularly alarming in the context of mental health advice, where the sensitive nature of the topic makes understandable guidance crucial. For instance, AIAN patients received mental health advice with Flesch reading ease levels as low as -8.7296, suggesting extremely difficult-to-read content. Conversely, White or European American (WEA), Asian (A), and Hispanic or Latino (HL) patients consistently received shorter, simpler advice.
The study also noted differences in how LLMs assessed medical emergencies, with advice for NHPI patients showing lower levels of perceived urgency, which is particularly concerning given the already complex language used for this group.
The Amplifying Effect of Intersectional Identities
A crucial insight from the study is how these disparities are greatly amplified when multiple socio-demographic factors intersect. The differences in advice length, readability, and sentiment were about twice as large when both sex and ethnicity were considered, compared to ethnicity alone. This underscores that examining individual categories in isolation can obscure significant inequities faced by those with intersectional identities.
Also Read:
- Public Increasingly Trusts AI Over Doctors for Medical Advice, Studies Reveal
- Generative AI-Integrated Counseling: A New Frontier for Paternal Support in Neonatal Intensive Care
Moving Towards Equitable AI Healthcare
Given the increasing trust individuals place in AI models for health guidance, the authors advocate for greater AI literacy among the public and an urgent call to action for AI developers. They emphasize the need to investigate and mitigate these systemic differences to ensure that AI-powered medical support is just and equitable for all.
The paper suggests several mitigation strategies, including fostering interdisciplinary collaboration between AI researchers, ethicists, and medical experts during model development. It also highlights the importance of incorporating local and culturally grounded knowledge into pre-trained data to reduce biases and create more reliable and equitable AI technologies.
While the study acknowledges limitations, such as the granularity of ethnic categories used, it paves the way for future work, including qualitative evaluations of advice substance and accuracy, and conducting focus groups with individuals from intersex, nonbinary, and ethnic minority communities to understand their perceptions of AI in healthcare.
