TLDR: A study investigated how Large Language Models (LLMs) align with human judgments on responsible AI values. It found that LLMs generally align more with AI practitioners than the general public, prioritizing fairness, privacy, transparency, safety, and accountability. However, a significant gap exists between the values LLMs *claim* to uphold and how they *actually prioritize* software requirements, highlighting inconsistencies in their applied ethical behavior and the critical need for human oversight in AI-assisted software development.
Large Language Models (LLMs) are rapidly becoming integral to various software engineering tasks, from defining requirements to evaluating designs. This widespread adoption raises a crucial question: how well do these advanced AI systems align with human judgments regarding responsible AI values? A recent study delves into this very topic, comparing LLMs’ value preferences with those of both a US-representative human sample and AI practitioners.
The research, titled “Are We Aligned? A Preliminary Investigation of the Alignment of Responsible AI Values between LLMs and Human Judgment,” explores this alignment across four distinct tasks. These tasks were designed to progressively test LLMs’ understanding and application of responsible AI principles. First, LLMs were asked to select the most important responsible AI values. Second, they rated the importance of these values in specific contexts, such as medical, banking, marketing, and streaming. The third task involved resolving trade-offs between competing values, like balancing performance with privacy. Finally, and perhaps most critically, the LLMs were tasked with prioritizing software requirements that embody these values, simulating real-world software development decisions.
The findings reveal a nuanced picture. Generally, LLMs demonstrated a stronger alignment with the perspectives of AI practitioners compared to the broader US-representative sample. Values such as fairness, privacy, transparency, safety, and accountability were consistently emphasized by both LLMs and AI practitioners. For instance, in the initial task of selecting key values, LLMs and AI practitioners both ranked fairness as highly important, a contrast to the US-representative sample which prioritized performance more.
When evaluating the importance of values in different contexts, LLMs tended to assign higher importance to responsible AI values overall. In medical and banking contexts, values were often rated as ‘extremely important’. However, variability emerged in marketing and streaming contexts, where human autonomy was not considered ‘extremely important’, and safety and fairness sometimes ranked lower. This contextual sensitivity, while present, also showed some differences compared to human judgments.
In resolving value trade-offs, LLMs largely mirrored human tendencies, consistently favoring safety over human autonomy and transparency. They also generally prioritized privacy and fairness above performance. However, some models showed indecision, particularly when balancing fairness with privacy or performance, indicating areas where their “moral compass” might be less clear-cut.
A significant insight from the study emerged from the fourth task: prioritizing high-level software requirements. Here, inconsistencies appeared between the values LLMs claimed to uphold in the abstract tasks (T1-T3) and how they actually prioritized requirements in a practical scenario (T4). For example, while fairness was often identified as a most important value abstractly, its priority could be reduced during concrete decision-making tasks. Similarly, safety, which might not be deemed extremely important in abstract terms for marketing or streaming, gained high priority when applied to specific requirements. This highlights a “faithfulness gap” – a discrepancy between stated preferences and applied behavior.
The researchers suggest that this misalignment could stem from several factors. The first type of misalignment, between LLMs and the general public, might be influenced by the cultural values embedded in training data and the selection biases in Reinforcement Learning from Human Feedback (RLHF) pipelines, which often draw from skilled annotators or domain experts, thus reinforcing practitioner perspectives. The second type, the inconsistency between stated preferences and actual behavior, echoes prior findings that LLMs can struggle with faithfulness, producing unfaithful explanations or inconsistent reasoning chains.
The implications for software engineering are profound. While LLMs offer significant potential for tasks like requirements elicitation and prioritization, this study underscores the need for caution. The observed inconsistencies mean that relying solely on LLMs for ethically grounded outcomes without human oversight carries practical risks. LLMs should currently serve as assistive tools, with requirements engineers actively involved in guiding, tuning, and reviewing their outputs to ensure alignment with diverse ethical priorities. This will require multidisciplinary collaboration among AI developers, requirements engineers, and end-users to systematically embed a broader range of values into AI-assisted development processes. For more detailed information, you can access the full research paper here.
Also Read:
- New Benchmark Reveals LLMs Struggle to Grasp Deep Human Values, Favoring Surface Preferences
- AI’s Cultural Compass: Navigating Global Values
The study also touched upon characteristics influencing LLM behavior, noting that smaller models showed higher variability. However, no single characteristic like size, release order, or family could reliably predict an LLM’s value system, indicating the complexity of achieving consistent ethical alignment.
