TLDR: A new research paper highlights the significant threat of “analytic flexibility” when using large language models (LLMs) to simulate human data, known as silicon samples. The study demonstrates that numerous methodological choices—such as the LLM model, temperature settings, demographic information provided, and prompting strategy—can dramatically alter the quality and consistency of simulated results. Critically, no single configuration consistently performs best across different evaluation metrics, implying that researchers must carefully consider and interrogate their analytic decisions to avoid generating unreliable or misleading findings in social science research.
Large language models (LLMs) are rapidly changing how social scientists conduct research, offering a new way to create “silicon samples” – synthetic datasets that mimic human responses. This approach promises to revolutionize human subjects research by making it more efficient, diverse, and accessible, especially for hard-to-reach populations. Researchers could use LLMs to pilot studies, predict responses, and compare hypotheses more swiftly than traditional methods.
Early applications of silicon samples have shown promise. Studies have found that LLMs can reproduce human-like behavior in economic games, replicate results from psychological studies, and generate large-scale datasets for sensory judgments. Some research even indicates strong correlations between LLM and human responses in moral judgment tasks, voting behavior, and problem-solving. Furthermore, by providing specific demographic information, LLMs have been used to simulate diverse participant profiles, predicting social science experiment outcomes and public opinion with considerable accuracy.
However, the enthusiasm for silicon samples is tempered by significant concerns. Not all studies have reported positive results; some found that LLMs failed to adjust judgments to subtle semantic shifts in scenarios, unlike humans. Even when LLMs did approximate human responses, minor variations in prompt formatting could dramatically alter outcomes. This sensitivity to prompt features, along with less diversity in response distributions compared to humans, highlights a critical issue: analytic flexibility.
The Challenge of Analytic Flexibility
Analytic flexibility refers to the numerous methodological choices researchers must make when generating silicon samples. These choices, which can be individually defensible, include the specific LLM used, its hyperparameter settings (like ‘temperature’ or ‘reasoning effort’), the strategy for presenting survey items (all-at-once, scale-by-scale, or item-by-item), and the amount of demographic information provided. The impact of these decisions on the quality of the synthetic data is often poorly understood.
This paper, titled “The threat of analytic flexibility in using large language models to simulate human data: A call to attention”, directly investigates this problem. The author conducted a study examining 252 different configurations of analytic decisions, using nine different LLMs and varying temperature/reasoning effort, demographic information, and scale sampling strategies. The goal was to see how these choices affected the consistency of silicon samples with real human data from 85 participants on two psychological scales: the Belief in a Just World (BJW) scale and the Gut Feelings scale.
Key Findings: Inconsistent Quality and Unpredictable Outcomes
The study revealed substantial variation in the quality of silicon samples based on analytic choices. The capacity of LLMs to estimate (i) the rank ordering of participants, (ii) response distributions, and (iii) between-scale correlations varied dramatically across configurations. For instance, correlations between human and LLM scores for participant ranking were generally low, ranging from -0.27 to 0.36 for the BJW scale and -0.25 to 0.32 for the Gut Feelings scale. While providing extensive demographics sometimes led to slightly higher correlations, the overall values remained small.
When assessing response distributions using Wasserstein distance, LLM-generated distributions were consistently and measurably larger than those observed in human-to-human comparisons, indicating a poorer approximation of human response patterns. The estimated correlation between the two scales in the silicon samples also varied wildly, from -0.26 to 0.71, despite the average being close to the true human correlation of 0.26. Crucially, even minor changes in settings, such as adjusting the temperature parameter, could drastically alter these predictions.
Perhaps the most critical finding was the lack of consistency in quality across different data features. A configuration that performed well in one area (e.g., ranking participants) often performed poorly in another (e.g., estimating response distributions). There was no “one-size-fits-all” configuration that optimized accuracy across all dimensions. Interestingly, the performance of a configuration in modeling the distribution of one scale did not significantly correlate with its performance on another scale, suggesting a lack of generalizability even within the same study.
Also Read:
- Crafting Digital Lives: How AI Generates Realistic Smartphone Usage Data
- Unmasking AI: Evaluation Awareness Grows Predictably with Model Scale
Implications and a Path Forward
The consequences of this analytic flexibility are significant. Researchers using silicon samples for study design or sample size planning could be led to incorrect conclusions, potentially underpowering or overpowering their studies. This inefficiency undermines the very purpose of using LLMs to improve research. More alarmingly, if LLMs cannot reliably simulate typical research subjects, their use for vulnerable or hard-to-reach populations risks creating an illusion of representation while potentially misportraying or flattening identity groups.
The paper argues that the social sciences are “sleepwalking into a literature filled with results that are strongly influenced by arbitrary methodological decisions rather than substantive underlying phenomena.” To prevent this, the author calls for greater attention to these analytic choices. Lessons from past mistakes in other scientific domains, such as the use of specification curve analysis and study registration (preregistration), can guide better practices.
Specification curve analysis, which visualizes the heterogeneity of results across different configurations, can help researchers identify which analytic dimensions have the most significant impact. Preregistration can ensure transparent documentation of analytic choices, distinguishing between initial testing and final decision-making. Ultimately, the goal is not just to document choices but to actively interrogate how they shape outcomes, aiming to identify classes of strategies that perform reliably well in specific contexts. Silicon samples hold immense potential, but realizing it requires a careful, thoughtful, and disciplined development process that takes nothing for granted.
