TLDR: Current “offline” evaluations of large language models (LLMs) fail to capture their real-world behavior because they don’t account for personalization, where models adapt to individual users. A new study by Angelina Wang, Daniel E. Ho, and Sanmi Koyejo shows that personalized “field evaluations” yield significantly different and more heterogeneous responses than offline tests, even impacting benchmark scores and model rankings. The researchers advocate for incorporating simulated user evaluations and greater transparency from LLM developers to create more realistic and accurate assessments of model performance and safety.
The way we currently evaluate large language models (LLMs) might not accurately reflect how they behave in the real world. A new research paper titled “The Inadequacy of Offline LLM Evaluations: A Need to Account for Personalization in Model Behavior” by Angelina Wang, Daniel E. Ho, and Sanmi Koyejo highlights this crucial disconnect.
Typically, LLMs are tested using “offline evaluations,” where they answer questions one at a time, without any memory of past interactions. This is like asking a fresh question to a brand new model every single time. However, in practice, users interact with LLMs through personalized interfaces like OpenAI’s ChatGPT or Google Gemini, which remember past conversations and user preferences. This is what the researchers call “field evaluation.”
The paper provides strong evidence that these two evaluation methods yield significantly different results. For instance, the same question posed to the same language model can produce very different answers depending on whether it’s a stateless system or a personalized chat session. This phenomenon was famously demonstrated with Microsoft Tay in 2016, a chatbot that quickly started producing harmful content after interacting with users, largely due to personalization effects not accounted for in its initial evaluation.
To demonstrate this, the researchers conducted a study involving 800 real users of ChatGPT and Gemini. They compared the outcomes of offline evaluations (using API calls) with field evaluations (users interacting with their personalized chat interfaces). They also introduced “sock puppet” evaluations, which simulate personalization in an offline setting by prepending user interaction history or profile descriptions.
One key finding from the study focused on recommendation questions (e.g., for haircuts, movies, restaurants). Field evaluations consistently showed more diverse and varied responses compared to offline evaluations. For example, when asked for company recommendations, offline evaluations recommended Tesla 93% of the time, while field evaluations diversified, recommending Tesla only 35% of the time. This suggests that personalization leads to a broader range of outputs. Among the simulated personalization methods, the “Profile” method, where the LLM is given a user description, produced the highest heterogeneity, sometimes even exceeding real-world field evaluations. This indicates that synthetic user profiles could be a promising way to simulate realistic variability.
The study also looked at benchmark questions from datasets like MMLU (measuring world knowledge) and ETHICS (measuring morality). For MMLU questions, field evaluations again showed greater response heterogeneity, even eliciting answer choices not seen in offline or sock puppet evaluations. This means that standard benchmarks might not capture the full spectrum of a model’s behavior when personalized.
Furthermore, the researchers explored the impact on MMLU scores. They found that personalization-induced variance in scores was significant enough to potentially reorder model rankings on leaderboards. For example, the performance gap between the top two models on the HELM Lite leaderboard is 0.6 percentage points, while the variability observed within their sock puppet evaluations was comparable to the 3.7 percentage point difference between the first and fifth-ranked models. This implies that current leaderboards, based on offline evaluations, might not accurately reflect how models perform in real-world personalized settings.
The authors argue that these findings have serious implications. If we benchmark models only in the typical offline fashion, we might not truly understand how they will perform in practice. This could lead to misleading safety assessments or inaccurate predictions of user experiences.
To address this, the paper proposes two main recommendations:
Also Read:
- The Hidden Flaws in AI Evaluation: Why LLM Judge Benchmarks Need a Rethink
- Unlocking Reliable AI Evaluation: How LLMs Can Judge More Effectively by Referencing Themselves
Recommendations for Future LLM Evaluation
1. Include “sock puppet” (simulated user) evaluations in benchmark assessments, as they better reflect user behavior than conventional offline studies. Researchers can use the study’s field evaluation methodology and data to validate their own simulated methods.
2. Organizations developing LLMs should provide researchers with access to anonymized or synthetic user profiles and transparency about personalization mechanisms. This would enable the creation of more realistic evaluations.
The paper emphasizes that personalization, often seen as just a product feature, is actually a fundamental requirement for any evaluation framework aiming to accurately reflect real-world LLM behavior. Ignoring this dimension can fundamentally mischaracterize model capabilities and deployment risks. You can read the full research paper for more details at this link: The Inadequacy of Offline LLM Evaluations.
