TLDR: New research reveals that large language models (LLMs) can spontaneously develop novel social biases against artificial demographic groups, even when no inherent differences exist. This phenomenon, stemming from an ‘exploration-exploitation trade-off,’ leads to highly stratified task allocations, often more severe than those by humans. Newer and larger LLMs show increased stratification. While some interventions like providing more context help, explicitly incentivizing diversity through prompt steering is most effective, highlighting the need for multifaceted objectives in AI development to mitigate emergent biases.
Large language models (LLMs) are increasingly integrated into systems that make real-world decisions, making it crucial to ensure their fairness. While much attention has been paid to removing existing biases that LLMs mirror from human data, new research suggests a more complex challenge: LLMs can spontaneously develop novel social biases on their own.
A recent study by Addison J. Wu, Ryan Liu, Xuechunzi Bai, and Thomas L. Griffiths from Princeton University and the University of Chicago explores this emergent bias. Their paper, titled “LARGE LANGUAGE MODELS DEVELOP NOVEL SOCIAL BIASES THROUGH ADAPTIVE EXPLORATION,” highlights that simply debiasing models isn’t enough. The researchers demonstrate that LLMs can create new stereotypes about artificial demographic groups, even when there are no inherent differences between these groups. These biases lead to highly unequal task allocations, often performing worse than human participants in terms of fairness.
The mechanism behind these emergent biases is rooted in what social scientists call the “exploration-exploitation trade-off.” This concept, also found in reinforcement learning, describes how decision-makers balance trying new options (exploration) with sticking to what has worked before (exploitation). The study suggests that LLMs, like humans, can explore too little, allowing early, random observations to disproportionately influence their impressions of entire demographic groups. This leads to a premature “lock-in” on perceived patterns, even if those patterns are spurious.
To illustrate this, the researchers adapted a “hiring game” paradigm from psychology. In this game, participants (or LLMs) act as hiring managers, assigning candidates from four artificial demographic groups (Tufa, Aima, Reku, Weki) to various jobs. Crucially, all candidates have an equal chance of success. However, initial random successes or failures can lead decision-makers to form inaccurate stereotypes, resulting in stratified job assignments where certain groups are consistently funneled into specific job types.
The findings were striking: not only did LLMs develop new biases, but frontier models (newer and larger LLMs) exhibited even greater stratification than human participants. This suggests a concerning trend where increased reasoning capabilities in LLMs might inadvertently lead to more unequal outcomes. The biases observed were also highly stochastic, meaning they were learned during each run of the experiment rather than being pre-existing in the models’ training data.
The research team investigated several interventions to mitigate these emergent biases. System-level changes, such as adjusting model temperature or using Chain-of-Thought (CoT) prompting, showed only marginal reductions in bias. Structural interventions, like lowering success probabilities or providing more contextual information about candidates (e.g., age and education in a refugee resettlement task), were more effective but often required unrealistic conditions or the availability of highly salient features. Interestingly, the effectiveness of additional features depended on their perceived contextual importance; arbitrary features like hair color or tattoo shape were less impactful.
The most robust and effective intervention involved explicitly incentivizing diversity through prompt steering. When the LLMs’ objective function was modified to include a measurable diversity bonus, they made significantly fairer and more equal allocations. This highlights that LLMs are powerful optimizers, and their objectives must be carefully formulated to align with societal values. The study emphasizes the need for “holistic objectives” that go beyond simple task completion to ensure AI systems remain unbiased as they interact with and shape the world.
Also Read:
- Uncovering Anchoring Bias in Large Language Models
- PRIME: A New Framework to Diagnose AI’s Stereotypical Reasoning
This research raises urgent questions about the long-term societal impact of LLMs. It reveals that these models are not just passive mirrors of existing human biases but can actively create new ones through their adaptive exploration. As AI systems become more agentic and stateful, evaluating their long-term influence in continuous, real-world contexts becomes paramount. For more details, you can read the full research paper here.
