TLDR: A study evaluated state-of-the-art LLMs’ ability to predict individual economic decisions using Pay-What-You-Want experiments with 522 real human personas. While LLMs showed reasonable group-level behavioral tendencies, their individual-level prediction accuracy was very low (under 5%). The research also found that structured persona formats (survey) outperformed narrative formats, and advanced prompting techniques offered limited improvements over basic methods, highlighting fundamental limitations in LLMs’ capacity to truly simulate complex human economic behavior.
Large Language Models (LLMs) have shown remarkable capabilities in simulating human-like behaviors, sparking significant interest across various fields. However, a common limitation in many studies is their reliance on fictional personas rather than actual human data. A recent research paper titled “Pay What LLM Wants: Can LLM Simulate Economics Experiment with 522 Real-human Persona?” addresses this gap by rigorously evaluating LLMs’ ability to predict individual economic decision-making using real human data.
Authored by Junhyuk Choi, Hyeonchu Park, Haemin Lee, Hyebeen Shin, Hyun Joung Jin, and Bugeun Kim, this study delves into how well LLMs can replicate complex human choices in a Pay-What-You-Want (PWYW) pricing experiment. The researchers utilized detailed persona information from 522 Korean participants involved in cultural consumption scenarios, providing a robust dataset grounded in real-world behavior.
The Experiment: Pay-What-You-Want Scenarios
The core of the study revolves around the Pay-What-You-Want (PWYW) pricing mechanism, where consumers voluntarily decide how much to pay for a product or service. This method is particularly insightful for economic decision-making as it involves intricate psychological factors like willingness to pay, income, cultural values, and individual value systems. The experiment designed two scenarios: art exhibitions and music performances, asking participants three key questions: willingness to attend at a regular price, willingness to attend at a self-determined price, and the specific amount they would pay.
The 522 human personas included 65 detailed items, covering demographics, cultural attitudes, knowledge, education, institutional satisfaction, and social norms. This comprehensive data allowed for a deep analysis of how LLMs could simulate individual choices.
Evaluating LLM Performance
The study explored two main research questions. The first, RQ1, investigated whether LLMs could predict individual economic decisions under two interaction conditions: Sequential and Human-guided. In the Sequential condition, LLMs built upon their own previous responses, while in the Human-guided condition, actual human responses were fed back into the conversation history to inform subsequent questions. The second question, RQ2, analyzed how different persona formats (structured Survey vs. narrative Storytelling) and prompting methods (basic, Chain-of-Thought (CoT), Retrieval-Augmented Generation (RAG), and Few-shot Prompting) affected LLMs’ simulation abilities.
Three state-of-the-art multimodal LLMs were chosen for evaluation: GPT-4o, Llama-3.2-90B-Vision-Instruct, and Qwen2.5-VL-72B-Instruct. Performance was assessed at two levels: individual-level accuracy (how precisely LLMs matched individual human choices) and group-level tendencies (overall similarity in behavioral patterns).
Also Read:
- Assessing LLM Capabilities in Explaining Well-Being Concepts: A Deep Dive into AI’s Readiness
- AI Judging AI: A New Era for Language Model Evaluation
Key Findings and Limitations
The results revealed several critical insights. The Human-guided condition generally led to superior prediction performance compared to the Sequential condition, especially for subjective questions. This suggests that providing actual human responses as examples can help LLMs align their reasoning more closely with human judgment. However, despite this improvement, the overall individual-level accuracy across all models and conditions remained very low, typically below 5%. This indicates a significant struggle for LLMs to precisely replicate individual human choices.
At the group level, LLMs demonstrated reasonable behavioral tendencies, showing similarities to humans in how various persona variables influenced decisions. GPT-4o consistently showed the highest performance in both individual and group-level metrics. Interestingly, the structured Survey Format consistently outperformed the narrative Storytelling Format, suggesting that LLMs process structured information more effectively for this task. Furthermore, commonly adopted advanced prompting methods like CoT, RAG, and Few-shot prompting showed only limited improvements over basic prompting, failing to significantly boost overall prediction accuracy.
A notable finding was the consistently low Jaccard Index across all experimental conditions. This metric measures the overlap between the set of significant variables identified by LLMs and humans, and its low values suggest that LLMs fundamentally struggle to identify and weigh the same decision-making factors that humans consider important. This points to intrinsic limitations in current LLM architectures regarding complex, contextual human decision-making.
In conclusion, while LLMs can capture general group-level behavioral tendencies, they currently fall short in precisely simulating individual human economic choices, even with rich real-world data and advanced prompting techniques. The study emphasizes that achieving more human-like behavioral simulation may require fundamental advances in model architecture and training paradigms, rather than incremental improvements in prompt engineering. For more details, you can read the full research paper here.
