TLDR: SIMBENCH is the first large-scale, standardized benchmark to evaluate how well large language models (LLMs) simulate human behavior across 20 diverse datasets. It finds that current LLMs have limited simulation ability (top score 40.8/100), performance scales with model size, and instruction-tuning improves consensus tasks but harms diverse opinion tasks. LLMs struggle with specific demographic groups, and simulation ability correlates strongly with knowledge-intensive reasoning. The benchmark is publicly available to drive progress in developing more faithful LLM simulators.
Large language models (LLMs) are increasingly being explored for their potential to simulate human behaviors, offering a faster and more cost-effective alternative to traditional human experiments and surveys. However, the current methods for evaluating how well these models mimic human actions are inconsistent and fragmented, making it difficult to compare results and understand their true capabilities.
To address this critical gap, researchers have introduced a groundbreaking new benchmark called SIMBENCH. This is the first large-scale, standardized tool designed to rigorously assess and compare the ability of LLMs to simulate group-level human behaviors. SIMBENCH unifies 20 diverse datasets, covering a wide array of tasks from moral decision-making to economic choices, and draws from a vast global participant pool. This comprehensive approach provides a solid foundation for understanding when, how, and why LLM simulations succeed or fail.
The initial findings from SIMBENCH reveal that even the most advanced LLMs today have limited simulation ability, with the top-performing model achieving a score of only 40.80 out of 100. This indicates that while promising, current LLMs are still far from perfectly replicating human behavior across diverse contexts. The study also found that simulation performance generally improves as model size increases, following a log-linear scaling law. Interestingly, simply increasing the computational effort during the model’s inference (test-time compute) does not significantly improve its simulation capabilities.
A crucial discovery is the “alignment-simulation trade-off.” Instruction-tuning, a common method to align LLMs with desired behaviors, improves performance on questions where humans largely agree (low-entropy questions). However, it actually degrades performance on questions with diverse human opinions (high-entropy questions). This suggests that current alignment techniques might inadvertently reduce the model’s ability to capture the full spectrum of human responses. Furthermore, models particularly struggle when asked to simulate the behaviors of specific demographic groups, especially those defined by religious or ideological affiliations.
The research also established a strong correlation between simulation ability and deep, knowledge-intensive reasoning capabilities, as measured by benchmarks like MMLU-Pro (r=0.939). This suggests that accurately simulating human behavior requires a broad and profound understanding of the world, rather than just narrow, specialized skills.
SIMBENCH was meticulously created by combining data from major social and behavioral science repositories and key academic papers. The datasets were selected based on strict criteria, including large participant counts, permissive licensing, single-turn multiple-choice questions, and rich sociodemographic data. This process ensured both task diversity, covering decision-making, self-assessment, judgment, and problem-solving, and participant diversity, spanning over 130 countries.
The benchmark is divided into two main splits: SimBenchPop, which evaluates the ability to simulate broad human populations, and SimBenchGrouped, which focuses on simulating narrower demographic groups based on specific characteristics like age or gender. The evaluation uses a SIMBENCH score derived from Total Variation Distance, which measures how much more accurate an LLM’s predictions are compared to a uniform baseline.
Also Read:
- Unlocking Individual Thought: A New Benchmark for Language Models
- Unmasking AI Deception: A New Benchmark Reveals Vulnerabilities in Large Language Models
By making progress measurable and providing a standardized framework, SIMBENCH aims to accelerate the development of more faithful LLM simulators. The researchers emphasize the importance of responsible use, cautioning against relying on LLM simulations for tasks where harm is possible, given their current limitations. All components of SIMBENCH, including datasets and code, are publicly available to foster collaborative research and ensure reproducibility. You can find more details in the full research paper: SIMBENCH: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors.
