TLDR: A new benchmark called MOREBENCH evaluates how AI models reason through moral dilemmas, rather than just their final decisions. It uses 1,000 scenarios with expert-written criteria and also tests reasoning under five ethical frameworks. Findings show current AI struggles with logical moral reasoning and that general AI capabilities don’t predict moral competence, highlighting the need for process-focused evaluation for safer AI.
As artificial intelligence systems become increasingly integrated into our daily lives, making decisions with us and for us, a crucial question arises: how do these systems arrive at their conclusions, especially when faced with complex moral dilemmas? A new research paper introduces MOREBENCH, a groundbreaking benchmark designed to evaluate not just what decisions AI models make, but the intricate reasoning processes behind them.
The paper, titled “MOREBENCH: EVALUATING PROCEDURAL AND PLURALISTIC MORAL REASONING IN LANGUAGE MODELS, MORE THAN OUTCOMES,” was authored by a diverse team of researchers including Yu Ying Chiu, Michael S. Lee, Rachel Calcott, Brandon Handoko, Paul de Font-Reaulx, Paula Rodriguez, Chen Bo Calvin Zhang, Ziwen Han, Udari Madhushani Sehwag, Yash Maurya, Christina Knight, Harry Lloyd, Florence Bacus, Mantas Mazeika, Bing Liu, Yejin Choi, Mitchell Gordon, and Sydney Levine. Their work highlights a critical gap in current AI evaluation methods, which often focus on objective tasks like math or coding where answers are definitively right or wrong.
Moral dilemmas, however, are different. They often allow for multiple defensible conclusions, making them an ideal testbed for understanding an AI’s procedural reasoning. MOREBENCH addresses this by presenting 1,000 moral scenarios, each meticulously paired with a set of over 23,000 rubric criteria written by moral philosophy experts. These criteria assess essential elements that should be included (or avoided) when an AI reasons about a scenario, covering aspects like identifying moral considerations, weighing trade-offs, and providing actionable recommendations.
The benchmark is designed to test AI in two core roles: as a Moral Advisor, guiding humans through everyday dilemmas, and as a Moral Agent, making autonomous decisions in high-stakes situations. To further deepen the evaluation, the researchers also curated MOREBENCH-THEORY, a subset of 150 examples specifically designed to test whether AI can reason effectively under five major frameworks in normative ethics: Kantian Deontology, Benthamite Act Utilitarianism, Aristotelian Virtue Ethics, Scanlonian Contractualism, and Gauthierian Contractarianism.
The creation of MOREBENCH involved a rigorous process, recruiting 53 moral philosophy experts with PhDs or equivalent experience. These experts developed the scenarios and the detailed rubric criteria, ensuring objectivity, specificity, and atomicity for grading individual aspects of moral reasoning. The criteria are categorized into five dimensions: Identifying (relevant moral considerations), Clear Process (systematic reasoning), Logical Process (integrating considerations), Helpful Outcome (clarifying paths and implications), and Harmless Outcome (avoiding illegal or harmful advice). Each criterion is also assigned a weight, reflecting its importance to a perfect response.
The findings from MOREBENCH offer surprising insights into the current state of AI moral reasoning. Contrary to expectations from scaling laws, which suggest larger models always perform better, the largest models within certain families did not consistently achieve the highest scores on MOREBENCH. Furthermore, the study revealed that popular benchmarks for general AI capabilities, such as those for math, code, or general-domain reasoning, are poor predictors of an AI’s performance in moral reasoning. This suggests that moral competence is a distinct capability not easily transferable from other forms of intelligence.
While models generally performed well in avoiding harmful outcomes (scoring over 77% on average in the Harmless Outcome dimension), they struggled significantly with the ‘Logical Process’ dimension, averaging only 41.5%. This indicates a difficulty in integrating various moral considerations and making reasonable trade-offs between conflicting priorities. Performance also varied moderately in identifying relevant factors, making clear reasoning, and supporting helpful outcomes.
On the MOREBENCH-THEORY dataset, models performed best when reasoning under Utilitarian and Deontological frameworks, averaging 64.8% and 65.9% respectively. However, their performance varied markedly on Virtue Ethics and Contractarianism, suggesting that AI models are not equally adept at applying different moral frameworks, even when explicitly instructed to do so. This highlights potential challenges for users attempting to customize AI’s moral inclinations.
Also Read:
- Unmasking AI Deception: A New Benchmark Reveals Vulnerabilities in Large Language Models
- Assessing How Well Large Language Models Simulate Human Behavior with SIMBENCH
In conclusion, MOREBENCH represents a significant step forward in evaluating AI’s moral reasoning capabilities. By focusing on the ‘how’ rather than just the ‘what,’ this benchmark provides a crucial tool for developing safer, more transparent, and more human-aligned AI systems. For more details, you can read the full research paper here.
