TLDR: A new research paper introduces a novel method for evaluating fairness and ethics courses in computing education. By combining a justice-oriented scoring rubric with an LLM simulating multiple evaluator perspectives (instructor, department chair, institutional, external), the study analyzed 24 syllabi. Findings reveal varying perceptions of justice-oriented content among evaluators and highlight gaps in explicit justice discussions and accessibility. The research provides recommendations for improving curriculum design to foster more inclusive and justice-centered AI education.
As Artificial Intelligence (AI) continues to integrate into various aspects of our lives, from healthcare to finance and criminal justice, the need to educate future generations on its ethical implications, fairness, and justice has become paramount. Computing courses, especially those focusing on AI, machine learning (ML), and algorithmic design, play a crucial role in shaping how students understand and address these complex issues.
However, evaluating the effectiveness and depth of these courses, particularly through their syllabi, presents a significant challenge. Manual syllabus review is often time-consuming, inconsistent, and can be influenced by the individual biases of evaluators. To overcome these limitations, a recent research paper introduces an innovative approach: using a large language model (LLM) to conduct a multi-perspective and thematic evaluation of course syllabi.
A New Lens for Syllabus Evaluation
The paper, titled “A Justice Lens on Fairness and Ethics Courses in Computing Education: LLM-Assisted Multi-Perspective and Thematic Evaluation” by Kenya S. Andrews, Deborah Dormah Kanubala, Kehinde Aruleba, Francisco Enrique Vicente Castro, and Renata A Revelo, proposes a novel framework. This framework combines a specially designed justice-oriented scoring rubric with LLM-simulated evaluator roles. The rubric consists of 20 criteria, assessing both inclusive learning practices (like authorship diversity and accessibility) and substantive fairness education (such as contextualizing fairness metrics and addressing historical inequities).
The researchers prompted OpenAI’s GPT-4o-mini to simulate four distinct evaluator perspectives: an instructor, a departmental chair, an institutional reviewer, and an external evaluator. Each persona was given specific priorities and constraints, allowing the LLM to assess syllabi from varied viewpoints. This multi-perspective analysis aimed to uncover how justice-oriented content might be perceived differently depending on the evaluator’s role.
Key Insights from Multi-Perspective Analysis
The study revealed fascinating insights into how different stakeholders view course content. Instructors and internal evaluators showed strong agreement, often rating criteria like “Diversity of Assessment Style,” “Potential for Bias Mitigation Techniques,” and “Accessibility of Resources” higher. In contrast, departmental chairs often gave lower scores, particularly for “Accessibility of Resources” and “Perspective of Topics.” This divergence suggests that justice-oriented elements might be more visible to those directly involved in teaching and less prioritized at higher administrative levels.
While standardized practices like using an “Institutional Template” and strong “Department Affiliation” consistently received high scores, weaknesses were identified in the “Clarity of Course Goals” and the recognition of an “Instructor’s Status (Tenure).” This highlights a need for more explicit justice-oriented objectives and greater transparency regarding the positionality of instructors.
Comparing graduate and undergraduate courses, the study found that graduate courses generally scored higher across most criteria, especially in “Participation Activities” and “Department Affiliation.” This indicates more intentional design and clearer expectations at the graduate level, while undergraduate courses often introduce foundational fairness principles but may lack the depth needed for a full understanding of core principles.
Thematic Trends in Fairness and Ethics Education
Beyond individual evaluations, the LLM also conducted a thematic analysis to identify common trends across the 24 syllabi collected from various U.S. institutions. The most common main course readings included “Weapons of Math Destruction” by Cathy O’Neil, “Fairness and Machine Learning” by Barocas, Hardt, and Narayanan, and “Trustworthy Machine Learning” by Kush Varshney. The researchers noted that while diverse voices are included as supplemental readings, they are less frequently featured as main texts, underscoring a potential gap in representation.
Interestingly, the most frequently occurring justice-oriented terms were ‘learning,’ ‘fairness,’ ‘social,’ ‘ethics,’ ‘project,’ and ‘impact.’ The term ‘justice’ itself was not widely used, suggesting a potential area for more explicit integration into course language.
Institutional trends also varied: Predominantly White Institutions (PWIs) often focused on ‘fairness’ and ‘ethical uses of AI,’ with ‘justice’ being less explored. Hispanic-Serving Institutions (HSIs) emphasized ‘ethics,’ ‘fairness,’ and ‘social impact,’ aligning with their focus on social cohesion. Historically Black Colleges and Universities (HBCUs) highlighted ‘ethics’ (with an emphasis on marginalized communities), ‘social impact,’ and ‘community engagement,’ reflecting their mission to address inequalities and promote community activism.
Also Read:
- Assessing Creativity: A Look at AI vs. Human Grading in Design Thinking Education
- How AI Thinks Morally: A Deep Dive into the MOREBENCH Evaluation
Recommendations for a More Just AI Education
Based on their findings, the researchers offer several recommendations to enhance fairness and ethics courses in AI, ML, and algorithmic design:
- Include evaluators from marginalized backgrounds and across institutional and external levels.
- Encourage instructors to use explicit, justice-oriented language and practices in course objectives.
- Apply a justice-oriented rubric to assess and refine syllabi.
- Integrate explicit discussions of justice, which are often missing but provide critical perspectives.
This study demonstrates the significant potential of LLM-assisted evaluation to complement human review, offering a scalable way to diagnose strengths and gaps in justice-oriented criteria within AI ethics courses. It also underscores the ongoing importance of human calibration, institutional context, and a clear focus on justice-aware criteria to truly prepare students for an AI-driven world. You can read the full research paper here: A Justice Lens on Fairness and Ethics Courses in Computing Education.
