TLDR: The paper introduces “Persona-Augmented Benchmarking,” a method to evaluate Large Language Models (LLMs) by rewriting evaluation prompts using diverse, persona-based writing styles. It finds that LLM performance is highly sensitive to these stylistic variations, even when semantic content is identical. Certain styles, particularly those associated with lower education or older age, consistently lead to poorer performance across various models and tasks, highlighting a lack of robustness in current LLMs and limitations in existing standardized benchmarks.
Large Language Models (LLMs) have shown incredible abilities, but their performance can be inconsistent, especially when dealing with different types of users. A key reason for this inconsistency lies in how these models are typically evaluated. Current benchmarks often rely on standardized, formal writing, which doesn’t fully capture the rich variety of ways humans communicate.
This narrow focus means that LLMs, optimized on these limited benchmarks, might struggle when faced with more diverse or “non-standard” inputs. Researchers from Carnegie Mellon University and Amazon AWS set out to test this idea. They developed a new approach called Persona-Augmented Benchmarking to evaluate LLMs across a wider range of writing styles.
How They Did It: Persona-Based Prompting
The core of this research involves using “personas” to rewrite evaluation prompts. A persona is a one-to-three sentence description of a character, combining socio-demographic details (like native language, age, education level) and psychosocial attributes (like interests or occupation). These personas guide an LLM to rephrase existing benchmark questions and contexts in a style that reflects how different individuals might express the same information.
The process involved four main steps: first, creating a diverse set of persona descriptions; second, using these personas to rephrase benchmark examples; third, carefully checking that the rephrased examples still contained all the original, necessary information (called “entailment-checking”); and finally, evaluating various LLMs using these newly styled prompts. Crucially, the method ensured that the semantic content – the core meaning – remained identical, only the writing style changed.
Key Findings: Style Matters More Than You Think
The study yielded several significant insights:
- Increased Linguistic Diversity: The persona-augmented benchmarks indeed showed much greater linguistic variation compared to the original, standardized texts. This confirms that LLMs can effectively generate diverse writing styles when guided by personas.
- Performance Sensitivity: LLM performance was highly sensitive to these persona-induced writing style variations. Even with the exact same underlying information, changes in writing style and prompt formatting significantly impacted how well the LLMs performed. Performance changes ranged from 15% to a striking 80% for a single model across different persona subsets, depending on the task.
- Consistent High and Low Performers: The researchers identified specific writing styles that consistently led to either notably high or low performance across a wide range of models, regardless of their family, size, or release date. For instance, prompts rephrased to sound like they came from someone with “less than high school-educated” or “elderly” personas frequently triggered poorer performance. Conversely, more academic or technical language often led to better results.
- Impact on Rankings: While the models in this study were chosen to have widely varying capabilities, simulations showed that in more competitive scenarios, like public leaderboards where models have closer scores, these performance shifts due to writing style could drastically alter model rankings – by as much as 19 positions up or 14 positions down.
Also Read:
- How Persona Prompts Influence Large Language Model Security
- AI Evaluating AI: A Benchmark-Free Method for LLM Assessment
Implications for LLM Development and Evaluation
These findings highlight a critical limitation of current LLM benchmarks: they often don’t represent the full spectrum of human communication. This can lead to an overestimation of an LLM’s real-world performance, as models might be brittle when encountering diverse user inputs.
The research suggests that practitioners should consider not just task-specific performance but also the target user population when selecting LLMs. The consistent performance drops observed across different models indicate a systemic issue in how these systems are trained and optimized, often favoring standardized language patterns. This calls for new development practices that prioritize robustness to varied writing styles.
The Persona-Augmented Benchmarking pipeline offers a practical and scalable solution to improve the external validity of existing LLM assessments without requiring costly new human data collection. It helps uncover potential failure modes and allows for deeper analysis of which linguistic features LLMs prefer or struggle with. For more technical details, you can read the full paper here.
