TLDR: A University of Washington study found that ChatGPT exhibits ableist biases, consistently ranking job applications with disability-related credentials lower and fabricating ableist reasons for exclusion. While some biases can be mitigated with specific instructions, the findings raise significant concerns about the impact of generative AI on disabled applicants in hiring processes.
A recent study conducted by the University of Washington has brought to light a concerning issue: the popular generative AI chatbot, ChatGPT, demonstrates inherent ableist biases when evaluating job applications. The research, published on June 24, 2024, indicates that ChatGPT consistently ranks resumes that include disability-related honors and credentials lower than those with similar qualifications but no mention of disabilities. This revelation raises serious questions about the fairness and ethical implications of using AI in recruitment.
The study’s methodology involved using a publicly available curriculum vitae (CV) as a baseline. Researchers then created enhanced versions of this CV, incorporating awards and honors that implied various disabilities. Examples included a ‘Tom Wilson Disability Leadership Award’ or participation in a Diversity, Equity, and Inclusion (DEI) panel. The AI was tasked with ranking these applicants, and the results were stark. Across 60 trials, the original, non-disability-inclusive CV was ranked first in 75% of the cases.
Kate Glazko, a computer science and engineering graduate student and the lead author of the study, emphasized the urgency of these findings. “Ranking resumes with AI is starting to proliferate, yet there’s not much research behind whether it’s safe and effective,” Glazko stated. She further highlighted the dilemma faced by disabled job seekers: “For a disabled job seeker, there’s always this question when you submit a resume of whether you should include disability credentials. I think disabled people consider that even when humans are the reviewers.”
Beyond simply ranking applications lower, the study also uncovered that ChatGPT would ‘hallucinate’ ableist justifications for its decisions. These fabricated reasons often suggested that certain mental or physical illnesses would impede a candidate’s ability to perform the job. Glazko noted, “Some of GPT’s descriptions would color a person’s entire resume based on their disability and claimed that involvement with DEI or disability is potentially taking away from other parts of the resume.”
However, the research also offered a glimmer of hope. It was found that some of these ableist tendencies could be mitigated. By explicitly instructing ChatGPT not to be ableist and by feeding it disability justice and DEI principles through the GPTs Editor feature, the enhanced CVs managed to outperform the original more than half of the time. Nevertheless, the effectiveness of this mitigation varied depending on the specific disability implied in the CV.
Also Read:
- Australian Firms Prioritize Experienced Talent Amidst AI Surge, Entry-Level Roles Decline
- Goldman Sachs Economist Warns AI Poses Initial Threat to Gen Z Tech Employment
This is not the first instance of bias being identified in OpenAI’s chatbot. A Bloomberg investigation in March previously revealed that the GPT 3.5 model exhibited clear racial preferences in job candidate evaluations, not only replicating discriminatory hiring practices but also perpetuating stereotypes across both race and gender. In response to such findings, OpenAI has maintained that these tests do not accurately reflect the practical applications of their AI models in real-world workplace scenarios.
