TLDR: A new benchmark called ROLECONFLICTBENCH evaluates how large language models (LLMs) handle complex social dilemmas where different roles clash. The study found that while LLMs show some awareness of situational urgency, their decisions are largely driven by inherent biases towards certain social roles, demographics, and oversimplified values, rather than a nuanced understanding of context. This highlights a need for more socially responsible AI.
Humans frequently face what are known as role conflicts—situations where the demands of different social roles clash, making it impossible to fulfill all expectations simultaneously. For instance, imagine a judge who is also a grandparent. What if they are in the middle of delivering crucial jury instructions when they receive an urgent message that their grandchild is in crisis? Which role should take priority?
As large language models (LLMs) become increasingly integrated into our daily lives, assisting with everything from personalized advice to automated decision-making in critical sectors like healthcare, it’s vital to understand how these AI systems navigate such complex social dilemmas. Previous research has often evaluated LLMs in contexts with clear-cut ‘correct’ answers. However, real-world role conflicts are inherently ambiguous, demanding a sophisticated understanding of situational cues—what researchers call ‘contextual sensitivity’.
Introducing ROLECONFLICTBENCH
To address this critical gap, researchers have introduced ROLECONFLICTBENCH, a novel benchmark specifically designed to evaluate LLMs’ contextual sensitivity in these challenging social scenarios. This benchmark employs a sophisticated three-stage pipeline to generate over 13,000 realistic role conflict scenarios across 65 different social roles. The scenarios systematically vary the expectations associated with these roles (their responsibilities and obligations) and the urgency levels of the situations.
The generation process begins with ‘Expectation Generation,’ where common social expectations for various roles are curated. This is followed by ‘Situation Instantiation,’ where specific situations are created for each expectation, each assigned an urgency score from 1 (light, routine) to 3 (pressing demand). Finally, ‘Story Synthesis’ combines these elements into first-person narratives where the individual faces a direct conflict between two roles, but their final decision is left unstated.
LLMs’ Performance: A Lack of Contextual Sensitivity
The study evaluated 10 different LLMs, including models from OpenAI (GPT-4.1, GPT-4.1-mini) and Google (Gemini 2.5 Flash, Gemini 2.5 Flash-Lite), as well as open-source models like Qwen3 and OLMo2 families. The findings revealed a significant limitation: while LLMs showed some capacity to respond to contextual cues like urgency, this sensitivity was largely insufficient. Instead, their decisions were predominantly driven by powerful, inherent biases related to social roles rather than the specific situational information provided.
For example, the analysis showed that even when a role’s urgency was high, a strong underlying preference for certain roles often overrode the immediate context. This ‘role preference’ was a primary driver of the models’ choices. The study quantified these biases, revealing a dominant preference for roles within the Family and Occupation domains across most evaluated models. Furthermore, a clear prioritization of male roles and Abrahamic religions was observed.
Demographic Cues and Oversimplified Values
The research also explored how LLMs react to demographic cues. When models were prompted with the same scenario but asked to respond ‘As a {demographic attribute}’ (e.g., man, woman, White, Black, Asian, Hispanic), their choices became unstable and were improperly influenced by these single tokens. This suggests that instead of objectively reasoning from the situation, the models defaulted to stereotype-driven patterns associated with the user’s demographics.
Another critical finding was the models’ tendency to map social domains to a very narrow set of prosocial values. For instance, Family and Interpersonal roles were almost exclusively explained by ‘Benevolence,’ while ‘Occupation’ was narrowly tied to ‘Security.’ This rigid and oversimplified mapping indicates a failure to grasp the complex, pluralistic motivations that drive human decision-making in real-world social contexts.
Also Read:
- Unveiling Self-Preference: How Large Language Models Develop Human-Like Bias
- Beyond a Single Roll: Why Repetitions Are Key to Reliable LLM Evaluations
Implications for AI Development
The findings from ROLECONFLICTBENCH have significant implications for AI safety and alignment. They highlight that current LLMs lack a nuanced understanding of complex social contexts and are prone to inherent biases that can override situational logic. This underscores the urgent need to test LLMs in more intricate social scenarios beyond simple, prescriptive situations. The benchmark serves as an essential tool for diagnosing these limitations, paving the way for the development of more robust, equitable, and socially responsible AI models for decision-making systems, personalized agents, and social simulations.
