TLDR: A study involving 94 participants evaluated LLM responses to privacy-sensitive scenarios, finding that while users generally perceive responses as helpful and privacy-preserving, there’s significant disagreement among users on the same responses. Critically, proxy LLMs, often used for evaluation, show high internal consistency but only weak to moderate correlation with average human judgments and fail to capture the diverse range of human perceptions. This highlights the need for human-centered evaluations and improved, personalized proxy LLMs that can better reflect individual privacy and utility preferences.
Large Language Models (LLMs) have become indispensable tools for a myriad of daily tasks, from drafting emails to answering complex health and legal questions. As their adoption grows, so does the concern about how these powerful AIs handle sensitive personal information that users might share. Previous research has attempted to evaluate LLMs’ ability to protect privacy using benchmarks and other LLMs (referred to as ‘proxy LLMs’). However, these evaluations often overlooked the crucial perspective of real users and primarily focused on privacy without equally considering the helpfulness of the responses.
A recent study, titled User Perceptions of Privacy and Helpfulness in LLM Responses to Privacy-Sensitive Scenarios, delves into these overlooked aspects. Conducted by Xiaoyuan Wu, Roshni Kaushik, Wenkai Li, Lujo Bauer, and Koichi Onoue, the research aimed to understand how actual users perceive the privacy-preserving quality and helpfulness of LLM-generated responses in sensitive situations. It also investigated whether proxy LLMs could accurately estimate these human perceptions.
The Study’s Approach
The researchers conducted a user study involving 94 participants who evaluated 90 privacy-sensitive scenarios drawn from the PrivacyLens dataset. For each scenario, a response was generated by OpenAI’s ChatGPT-5 (GPT-5). Participants were asked to rate the helpfulness and privacy-preserving quality of these responses using a five-point scale and provide explanations for their choices. To compare human judgments with AI evaluations, five different proxy LLMs (Gemma-3, GPT-5, Llama-3.3, Mistral, and Qwen-3) were also tasked with completing the same survey for all 90 scenarios, with each proxy LLM evaluating every scenario five times to check for consistency.
Key Findings: A Divergence in Perception
The study yielded several significant insights:
Firstly, individual participants generally found the LLM-generated responses to be helpful and privacy-preserving most of the time. Over 90% of evaluations indicated the response completed the task, and 87% found the response helpful. Similarly, 78% of the time, participants felt the responses complied with privacy norms, and 83% believed they respected personal privacy preferences.
However, a critical finding emerged when comparing participants’ evaluations of the *same* scenario: users often disagreed significantly with each other. For instance, when evaluating an email about team morale, one participant found the response “very helpful” for its casual and approachable style, while another deemed it “very unhelpful” for missing key business details. Similar disagreements arose regarding privacy, with one participant finding a message about a school experience to “completely” follow privacy norms, while another felt it “not at all” respected privacy due to too much private information.
Secondly, the proxy LLMs exhibited high self-consistency across their multiple runs and moderate agreement with each other. This suggests that AI evaluators tend to produce uniform judgments. Yet, when these proxy LLM evaluations were compared to the *average* human judgments, the correlation was only weak to moderate. More importantly, proxy LLMs failed to capture the wide range of evaluations and diverse perspectives that real users demonstrated. For example, while participants never fully agreed on the sensitivity of information in a scenario, proxy LLMs showed consistent evaluations in over 50% of cases.
Qualitative analysis of the explanations provided by both participants and proxy LLMs shed light on these misalignments. Proxy LLMs sometimes overlooked crucial contextual details, such as the audience of a message or the specific nuances of a task. They also occasionally failed to identify highly sensitive information, like credit card details, or had different interpretations of what constitutes a privacy norm. For instance, a proxy LLM might consider sharing client names as “non-sensitive,” while human participants strongly disagreed, citing violations of professional privacy norms.
Also Read:
- Unmasking Privacy Risks: Membership Inference in Clinical AI Models
- AI’s Eye for Motives: How Language Models Learn to Discern Intent in Communication
Implications for the Future of LLM Evaluation
The study’s findings underscore a fundamental challenge: current proxy LLMs are not reliable substitutes for human judgment when evaluating privacy and helpfulness in sensitive scenarios. User perceptions are highly individualized and context-dependent, a complexity that AI evaluators currently struggle to grasp.
The researchers advocate for continued human involvement in evaluating LLM-generated content, especially in domains where subjective factors like privacy and utility are paramount. They also suggest avenues for improving proxy LLMs, such as exploring methods to expand their range of evaluations to better reflect human diversity, perhaps through advanced training or adjusting model settings. Personalizing proxy LLMs to align with individual users’ privacy and utility preferences is another promising direction.
Finally, the study calls for clearer distinctions between tasks where consistent, objective answers are desirable (e.g., factual questions) and those where diversity in responses is appropriate due to subjective preferences and perceptions (e.g., privacy and utility). Establishing such taxonomies can guide researchers in designing more effective evaluation metrics and help users understand when to trust or question LLM outputs.
In conclusion, while LLMs offer immense utility, their ability to navigate the intricate landscape of human privacy and helpfulness remains a complex challenge. Real user input is indispensable for truly understanding and improving how these powerful AI systems interact with our most sensitive information.
