TLDR: The paper introduces “Personalized Deep Research Bench,” the first benchmark to evaluate how well Deep Research Agents (DRAs) can tailor their research to individual user needs. It features 250 realistic user-task queries and a new PQR Evaluation Framework (Personalization Alignment, Content Quality, Factual Reliability). Experiments show open-source agents lead in personalization, commercial agents offer reliability, and LLMs with search tools lag behind. The study highlights the importance of explicit user profiles for effective personalization and the current limitations of memory systems in inferring user needs from context.
Deep Research Agents (DRAs) represent a significant leap in AI, capable of autonomously conducting complex investigations and generating comprehensive reports. These advanced systems, powered by large language models (LLMs), move beyond simple question-answering to perform dynamic reasoning, adaptive planning, and iterative tool use to gather, synthesize, and analyze information, ultimately producing structured and detailed research.
However, despite their impressive capabilities, a crucial aspect has largely been overlooked in their evaluation: personalization. Current benchmarks for deep research agents primarily focus on factual accuracy and comprehensiveness, neglecting how well these agents can adapt their outputs to the unique needs, preferences, and contexts of individual users. This gap means that while DRAs can generate extensive reports, their ability to act as truly personalized assistants, tailoring information filtering, reasoning, and recommendations, remains largely unassessed.
To address this critical oversight, a new research paper introduces the Personalized Deep Research Bench, the first benchmark specifically designed to evaluate personalization in Deep Research Agents. This innovative benchmark provides a rigorous framework to assess how effectively agents integrate user profiles into their research workflows, ensuring outputs are not only accurate and comprehensive but also highly tailored and practically useful for the end-user.
Constructing the Personalized Deep Research Bench
The benchmark is built upon two core components: a diverse set of deep research tasks and a collection of authentic, multifaceted user profiles. Researchers formulated 50 distinct deep research tasks across 10 varied domains, such as Education, Career Development, Healthcare, and Financial Planning. These tasks were designed in collaboration with domain experts and validated for complexity, clarity, and alignment with personalized research scenarios.
A key innovation lies in the creation of 25 highly realistic user profiles. Instead of synthetic characters, these profiles were constructed from 25 diverse volunteers, capturing structured persona attributes (like age, gender, occupation, preferences) and dynamic real-world contexts (simulated daily interactions, memory snippets, and conversations). By pairing these 50 tasks with the 25 user profiles, the benchmark yields 250 unique and realistic user-task queries, enabling a systematic evaluation of both task complexity and persona-driven adaptation.
The PQR Evaluation Framework
To assess system performance comprehensively, the paper proposes the PQR Evaluation Framework, which measures three orthogonal dimensions:
- Personalization Alignment (P): This dimension evaluates how well the report addresses the user’s explicit and implicit goals, the suitability of its content for their knowledge and interests, the alignment of its presentation style, and its actionability and practicality for decision-making. This is a dynamic evaluation, generating customized criteria and scores for each user-task pair.
- Content Quality (Q): This assesses the intrinsic quality of the generated report, focusing on its depth, insight, logical coherence, clarity, and readability, independent of user profiles.
- Factual Reliability (R): This measures the factual accuracy of claims and the coverage of citations within the report, ensuring the information provided is trustworthy and well-supported.
The overall score for a report is a weighted aggregation of these three dimensions, providing a holistic, user-centered assessment.
Key Findings from Experiments
Experiments were conducted on a diverse range of systems, including commercial deep research systems (like Gemini-2.5-Pro Deep Research), open-source deep research agents (such as OAgents), and leading LLMs equipped with search tools (e.g., GPT-4.1 w/Search). The evaluation, performed on a subset of 150 representative queries, revealed several important insights:
- Open-source Agents Excel in Personalization: Systems like OAgents achieved the highest personalization scores, demonstrating strong alignment with user goals and presentation preferences. However, they often showed weaknesses in factual reliability, with lower factual accuracy or citation coverage.
- Commercial Agents Offer Balanced Quality and Reliability: Commercial systems, such as Gemini-2.5-Pro Deep Research, provided a good balance of content quality and high factual reliability, with strong citation coverage. While reliable, they lagged moderately behind open-source agents in personalization.
- LLMs with Search Tools Fall Short: General LLMs augmented with search capabilities generally underperformed specialized deep research agents across personalization, quality, and reliability metrics. Simply adding a search tool was insufficient to match the performance of dedicated deep research systems.
The study also investigated the impact of information availability on personalization. It found that providing more user information (from task-only to task with context, and then to task with explicit persona) consistently improved personalization scores. Explicit user personas yielded the largest gains, indicating that while agents can leverage implicit context, they struggle to fully extract user preferences from unstructured data as effectively as from clear, structured profiles.
Furthermore, the research explored whether advanced memory systems could transform unstructured context into explicit personas to boost personalization. While some memory systems showed promising results, a significant gap remains between their performance and that achieved with explicit personas, highlighting a need for future research in higher-level reasoning and user information integration.
Also Read:
- VitaBench: A New Standard for Evaluating LLM Agents in Real-World Scenarios
- Unmasking the Vulnerabilities of AI Search Agents: A New Framework for Safety Assessment
Paving the Way for Truly Personalized AI
This work establishes a rigorous foundation for developing and evaluating the next generation of AI research assistants. By formally introducing the task of personalized deep research and providing a comprehensive benchmark and evaluation framework, the paper addresses a critical gap in the field. It paves the way for AI agents that can not only conduct complex investigations but also genuinely adapt to individual user needs, making AI research assistants more effective and truly personal. You can read the full research paper for more details at arXiv:2509.25106.
