TLDR: A study investigated how Large Language Models (LLMs) respond to survey questions when prompts are subtly changed. It found that LLMs, like humans, exhibit biases, particularly a strong “recency bias” where they favor the last answer option presented. Larger models were generally more robust, but all models were sensitive to certain changes, highlighting the need for careful prompt design when using LLMs for synthetic survey data.
Large Language Models (LLMs) are increasingly being used as stand-ins for human participants in social science surveys, offering a promising way to generate synthetic data. However, a recent study delves into a critical question: how reliable are these LLM responses, and do they exhibit the same biases seen in human surveys?
Researchers Jens Rupprecht, Georg Ahnert, and Markus Strohmaier from the University of Mannheim, GESIS – Leibniz Institute for the Social Sciences, and Complexity Science Hub, Vienna, conducted an extensive study to explore the robustness and potential biases of LLMs in normative survey contexts. Their work, detailed in their paper “Prompt Perturbations Reveal Human-Like Biases in LLM Survey Responses”, involved over 167,000 simulated interviews with nine different LLMs.
The Study’s Approach
The team used questions from the World Values Survey (WVS), a globally diverse survey on human beliefs and values. They systematically applied eleven different “perturbations” – subtle changes to both the question phrasing and the structure of the answer options. These perturbations were designed to mimic common textual variations and errors, as well as to test for known human survey response biases like response order effects or the presence of a “don’t know” option.
Each of the nine selected LLMs, varying in size and developer (including Gemini-1.5-Pro, various Llama models, Mistral, Phi-3.5-mini, Qwen2.5, and Yi-1.5), was interviewed 25 times with each original and perturbed question-and-answer pair. This rigorous approach allowed for a detailed analysis of how consistent and robust LLM responses were under different conditions.
Key Findings: Robustness and Consistency
The study revealed several important insights into LLM robustness:
-
Model Size Matters: Generally, larger LLMs like Llama-3.3-70B and Gemini-1.5-Pro demonstrated greater robustness, often replicating their original answers in over 50% of cases despite perturbations. Smaller models were significantly less robust.
-
Combined Perturbations are Challenging: The most significant impact on robustness came from combining two perturbations, such as a paraphrased question with reversed answer options. This suggests that multiple subtle changes can have a compounding effect.
-
Semantic vs. Lexical Changes: Paraphrasing an entire question reduced robustness more than simply replacing individual words with synonyms. This indicates that LLMs are more sensitive to changes in overall meaning or structure than to minor word substitutions.
-
Typographical Errors: Not all typos are equal. Randomly replacing characters (Key Typo) or using adjacent keys (Keyboard Typo) had a greater negative impact on robustness than simply swapping two letters within a word (Letter Swap).
-
Scale Length: The complexity of the task also played a role. Robustness decreased as the number of answer options increased, meaning LLMs were less likely to give consistent responses on a 10-point scale compared to a 4-point scale.
Evidence of Human-Like Biases
Beyond robustness, the study sought to determine if LLMs exhibit biases similar to those found in human survey respondents:
-
Recency Bias: A consistent and often substantial “recency bias” was observed across all nine models. When the order of answer options was reversed, LLMs disproportionately favored the last-presented option. For instance, one model showed a staggering 2000% increase in the probability of choosing a specific option when it was moved to the last position.
-
Opinion Floating and Central Tendency: The effects of removing a “Don’t know” option (opinion floating) or adding a middle category (central tendency) were highly model-dependent. Larger models were generally more robust to these changes, while some smaller models showed a tendency to shift responses towards the center when a refusal option was absent or when a middle category was explicitly provided.
-
Emotional Priming: An emotional priming statement (“This is very important to my research!”) had inconsistent effects. For larger models, it either had no effect or slightly decreased refusal rates, while for some Chinese models, it surprisingly increased refusal rates.
Also Read:
- Unpacking Cognitive Biases in AI-Generated Content: A New Study Reveals How LLMs Influence Users
- The Global Challenge: How Language Model Overconfidence Affects Users Across Diverse Languages
Implications for Synthetic Data Generation
The findings underscore the critical importance of careful prompt design and rigorous robustness testing when using LLMs to generate synthetic survey data. Researchers are advised to use larger LLMs for better consistency, prefer smaller answer option scales for reproducibility, and carefully consider the inclusion or exclusion of middle and refusal categories, as these can significantly influence LLM responses.
While LLMs offer a powerful tool for social science research, this study highlights that their responses are not always stable and can be influenced by subtle prompt variations, sometimes mirroring human-like biases. Understanding these vulnerabilities is crucial for ensuring the reliability and validity of synthetic survey data.
