TLDR: A new study investigates how large language models (LLMs) align with human social reasoning in collective decision-making. Using the ‘Lost at Sea’ task, researchers found that some LLMs ‘mirror’ human biases, reproducing gender-skewed leader selections, while others ‘mask’ these biases, leading to more optimal but less human-aligned outcomes. The study emphasizes that human-AI alignment depends on context, identity cues, and model-specific biases, highlighting a tension between accurate simulation and achieving ideal results.
Large language models (LLMs) are increasingly integrated into various aspects of our lives, including collective decision-making. This raises a crucial question: how well do these AI systems align with human social reasoning? A recent study titled To Mask or to Mirror: Human-AI Alignment in Collective Reasoning delves into this complex issue, moving beyond individual-level alignment to explore how LLMs behave in group settings.
The research, conducted by Crystal Qian, Aaron Parisi, Clémentine Bouleau, Vivian Tsai, Maël Lebreton, and Lucas Dixon, introduces an empirical framework to assess collective alignment. They utilized a well-known social psychology task called ‘Lost at Sea,’ which involves groups deliberating and electing a leader. This task is particularly useful because it has previously shown persistent gender biases in leader selection.
The study involved a large-scale online experiment with 748 human participants. These groups were randomly assigned to one of two conditions for leader elections: either with visible demographic attributes like name and gender (identified condition) or with pseudonymous, gender-neutral aliases (pseudonymous condition). Following the human experiment, the researchers simulated matched LLM groups, conditioning them on the human data. They benchmarked several prominent LLMs, including Gemini 2.5, GPT 4.1, Claude Haiku 3.5, and Gemma 3.
The findings revealed a fascinating divergence in LLM behaviors. Some models, like Gemini and GPT, tended to ‘mirror’ human biases. In the identified condition, humans elected male leaders 64% of the time, and these LLMs reproduced this pattern, including the magnitude and structure of the leadership gap. This gap indicated that the elected leader performed significantly lower than the most competent candidate, a suboptimality driven by both top candidates not nominating themselves (self-exclusion) and the group not selecting them even if they did (peer-exclusion).
In contrast, other models, notably Claude, acted as a ‘mask.’ Instead of reproducing human biases, Claude groups projected more meritocratic outcomes, meaning they selected more optimal leaders with a significantly smaller performance gap. However, this came at the cost of lower alignment with actual human group behaviors. When explicit demographic cues were removed in the pseudonymous condition, the human gender gap in leader selection reduced, and optimal leaders were chosen more frequently, primarily due to decreased peer-exclusion. Gemini continued to mirror human patterns in this condition, while Claude’s performance deteriorated, defaulting to aligning with male-elected leaders.
A critical insight from the study is that human-AI alignment in collective reasoning is highly dependent on context, the cues provided, and the specific inductive biases of each model. When demographic information was available, Gemini and GPT mirrored human social patterns, including biases. Claude, on the other hand, used these identity cues to compensate for bias, leading to more optimal selections. When all demographic context was removed from the LLM simulations, alignment with human decisions completely disappeared, underscoring that persona construction with identity cues is essential for effective social simulation.
Also Read:
- Unpacking AI’s Social Compass: How Language Models Navigate Role Conflicts and Reveal Hidden Biases
- Unveiling Self-Preference: How Large Language Models Develop Human-Like Bias
The research highlights a fundamental tension: whether LLMs should accurately simulate human behavior, including its flaws (mirroring), or attempt to correct for human biases and achieve more ideal outcomes (masking). The choice of model and the context in which it operates are therefore crucial design decisions for applications involving group dynamics, such as developing interventions to support optimal leader selection or creating benchmarks for collective reasoning. Understanding when LLMs reflect, suppress, or distort social dynamics is vital for advancing socially-aligned AI.
