TLDR: A new study explores how Large Language Models (LLMs) can act as ‘digital twins’ to replicate and predict human behavior in game theory experiments. By simulating classic dyadic games, researchers found that the Llama model accurately reproduced human cooperation patterns, capturing deviations from purely rational choices. In contrast, the Qwen model aligned more closely with Nash equilibrium predictions. The study also developed advanced answer extraction methods to improve LLM performance and used Llama to generate testable hypotheses for novel game scenarios, demonstrating the potential of LLMs to accelerate scientific discovery in social and behavioral sciences.
Large language models (LLMs) are becoming increasingly prevalent, not just in making decisions across fields like health and education, but also in simulating human behavior. A critical question, however, is how accurately these AI models reflect actual human decision-making. Misalignment could lead to undesirable outcomes in practical applications or render LLMs ineffective for social simulations. A recent research paper, “Large language models replicate and predict human cooperation across experiments in game theory”, addresses this gap by creating a digital twin of game-theoretic experiments and introducing a systematic framework for evaluating machine behavior.
The study, conducted by Andrea Cera Palatsi, Samuel Martin-Gutierrez, Ana S. Cardenal, and Max Pellert, aimed to assess whether LLMs could replicate empirically observed human patterns in classic game theory scenarios. This is a crucial first step before using LLMs to extend experiments to previously unexplored areas, potentially making the experimental process faster and more cost-effective.
The Digital Twin Experiment
The researchers based their LLM setup on existing human experiments involving over five hundred participants playing dyadic games. In these games, two players simultaneously choose between cooperating (C) or defecting (D), with payoffs determined by both choices. The core of the experiment involved a payoff matrix where the reward for mutual cooperation (R) was fixed at 10 points, and the punishment for mutual defection (P) was 5 points. The ‘sucker’s payoff’ (S) and ‘temptation to defect’ (T) values were varied, creating 121 distinct games. These games encompassed four classical types: Harmony Game, Snowdrift Game, Stag Hunt Game, and Prisoner’s Dilemma.
For their digital twin, the team extended the parameter space for S and T, simulating 441 games in total, significantly more than the original human experiments. They used three open-source LLMs: Llama-3.1-8B-Instruct (referred to as Llama), Mistral-7B-Instruct-v0.3 (Mistral), and Qwen2.5-7B-Instruct (Qwen), chosen for their balance of performance, computational feasibility, and architectural diversity.
Unlocking Human-like Behavior: The Role of Extraction Strategies
A significant methodological contribution of this research was the development of progressively complex answer extraction methods. Initially, simple extraction of one-word answers from LLMs often yielded random-like responses. The researchers found that allowing models to generate longer, detailed explanations (Double Extraction) led to more logically coherent reasoning. Further improvements came from a Multi-step Extraction prompt, which guided models through logical reasoning steps. The most effective method, Logical Verifier Extraction, added a quality control layer where a separate LLM (Qwen) filtered responses for logical validity and mathematical accuracy before extracting the final choice.
This layered approach dramatically reduced noise and revealed cooperation patterns in Llama that closely resembled human empirical data. The study highlights that simply asking an LLM for a direct answer might not reveal its true reasoning capabilities; instead, structured prompting and verification are key to eliciting robust and meaningful behavioral responses.
LLMs and Human Behavior: A Tale of Three Models
The study found distinct behavioral characteristics across the three LLMs:
-
Llama: This model most closely mirrored human cooperation patterns, showing similar qualitative features in its cooperation matrix. It exhibited high cooperation when S ≥ T (Harmony Games) and lower cooperation when T > R, reflecting sensitivity to the temptation payoff, much like humans.
-
Qwen: Qwen’s behavior aligned closely with the Nash equilibrium, which represents rational economic behavior. This suggests Qwen might excel at strategic reasoning, potentially due to its training data or superior logical capabilities.
-
Mistral: Mistral displayed an intermediate pattern, showing sensitivity to payoff structures but not fully optimizing in a game-theoretic sense, nor perfectly replicating human patterns.
Quantitatively, Llama showed the highest similarity to human data (Mean Squared Displacement = 0.031, Pearson’s r = 0.89), while Qwen was most similar to the Nash equilibrium (MSD = 0.036, r = 0.93).
Beyond Replication: Predicting Novel Human Behaviors
A key innovation of this research is using Llama, the model most aligned with human behavior, to simulate novel game configurations outside the original human-tested parameter grid. These simulations generated empirically testable hypotheses about how humans might behave in unexplored strategic settings. For instance, Llama consistently showed high cooperation when S ≥ T, extending this pattern into the enlarged cooperation matrix.
This approach offers a complementary path to traditional social and behavioral science research. Instead of solely replicating known patterns, LLMs can be used to systematically explore vast experimental spaces, generating predictions that can then be validated through future human experiments. The researchers have even preregistered the precise experimental setup for these future studies, ensuring transparency and rigorous hypothesis testing.
Also Read:
- Large Language Models Uncover and Idealize Human Psychological Structures
- Decoding Human Choices: A New Framework for Personalized Decision Modeling
Implications for Understanding Human Decision-Making
The contrast between LLM-based predictions and the analytical Nash equilibrium is particularly insightful. While Nash equilibrium provides a clear theoretical rationale for rational choices, it often fails to reproduce observed human cooperation patterns. Llama’s closer alignment with human behavior suggests that LLMs may capture the psychological and heuristic processes underlying human decision-making, which often deviate from pure rationality. This aligns with the theory of bounded rationality, acknowledging humans’ limited information access and computational capacities.
The study also highlights that persona-based prompting, which attempts to simulate individual personalities, was not necessary to replicate aggregate behavioral patterns. Focusing on population-level trends proved to be a more tractable and robust approach for understanding collective human behavior.
In conclusion, this research demonstrates the significant potential of appropriately calibrated LLMs as digital twins in the social and behavioral sciences. By replicating human cooperation patterns and extending simulations to novel scenarios, LLMs can accelerate scientific discovery, offering a powerful tool for generating new empirical predictions about human social decision-making.
