TLDR: A new framework, DEEPRESEARCH-REPORTEVAL, is introduced to evaluate advanced AI research agents by assessing their generated reports. It measures quality (comprehensiveness, coherence, clarity, insightfulness), redundancy, and factuality using an LLM-as-a-Judge methodology validated against human experts. Experiments on commercial systems reveal trade-offs between report length, analytical depth, and factual grounding, with Qwen and OpenAI showing strong performance in evidence-based credibility.
Artificial intelligence is constantly evolving, and one of the most exciting new developments is the emergence of DeepResearch agents. These advanced AI systems are designed to perform expert-level research, much like a human expert would, by combining sophisticated reasoning with the ability to use multiple tools. Unlike traditional AI models that might just answer questions, DeepResearch agents can gather information from various sources, generate new insights, and present their findings in comprehensive reports.
However, evaluating these powerful systems has been a significant challenge. Traditional methods for assessing AI often focus on isolated capabilities, like how well a language model can hold a conversation or retrieve a specific fact. But DeepResearch agents do much more; they synthesize diverse information and produce coherent findings, which are difficult to measure with simple checks. To address this, researchers have introduced a new framework called DEEPRESEARCH-REPORTEVAL.
Introducing DEEPRESEARCH-REPORTEVAL
The DEEPRESEARCH-REPORTEVAL framework is specifically designed to evaluate DeepResearch systems by looking at their most important output: the research reports they generate. The creators of this framework, including Tianyu Fan, Xinyao Niu, Yuxiang Zheng, Fengji Zhang, Chengen Huang, Bei Chen, Junyang Lin, and Chao Huang, recognized that these reports are the primary way users interact with and assess the capabilities of these AI agents. The framework systematically measures three key aspects of these reports: quality, redundancy, and factuality.
The evaluation uses an innovative “LLM-as-a-Judge” methodology, where a large language model is trained to assess the reports, achieving strong agreement with human experts. This ensures that the automated evaluation is reliable and credible. The framework also includes a standardized benchmark of 100 carefully selected queries across 12 real-world categories, allowing for a consistent comparison of different DeepResearch systems.
How Reports Are Evaluated
The framework breaks down report evaluation into three main dimensions:
Quality: This dimension assesses several aspects of a report’s excellence. It looks at comprehensiveness (whether all key aspects are covered in depth), coherence (how well-organized and logically flowing the report is), clarity (the fluency and accuracy of the language), insightfulness (whether the report offers deep understanding and fresh analytical viewpoints beyond simple information compilation), and overall quality (a holistic assessment). Each aspect is scored from 0 to 4, with detailed justifications provided by the LLM-as-a-Judge.
Redundancy: DeepResearch systems often synthesize information from many sources. Sometimes, this can lead to repeating content or overlapping discussions, which inflates report length without adding much value. The framework tackles this by segmenting reports into paragraphs, then assessing the semantic independence of all possible paragraph pairs. A score from 0 to 4 indicates the degree of redundancy, with 4 meaning almost no repetition.
Factuality: Research reports must be grounded in factual evidence and include accurate citations. This evaluation checks whether claims made in the report are genuinely supported by their cited sources. It extracts claim-source pairs and uses the LLM-as-a-Judge to determine if the source fully supports, partially supports, or does not support the claim. Two metrics are used: Average Support Score and Strong Support Rate, to quantify the factual accuracy and reliability of the claims.
Key Findings from System Evaluation
The researchers evaluated four leading commercial DeepResearch systems: OpenAI, Perplexity, Gemini, and Qwen. The findings revealed distinct design philosophies and performance trade-offs among these systems. For instance, Perplexity generated significantly shorter reports, prioritizing conciseness and readability, which led to high scores in coherence and clarity but lower scores in comprehensiveness and insightfulness. This highlights a trade-off between brevity and analytical depth.
OpenAI and Qwen produced reports of moderate length (around 5000-7000 tokens), while Gemini’s reports were the longest. The study suggested that there might be an optimal report length, as quality doesn’t necessarily increase linearly with length. Qwen emerged as a top performer in both comprehensiveness and insightfulness, demonstrating that it can maintain analytical depth while still being readable. In terms of factuality, Qwen and OpenAI were found to be the most evidence-grounded systems, providing a higher number of claim-source pairs and achieving strong support scores, indicating their sophisticated ability to substantiate claims with cited sources.
Also Read:
- AutoQual: An LLM Agent for Automated Discovery of Interpretable Features for Review Quality Assessment
- Enhancing Survey Insights: A New Framework for Evaluating Human Responses
The Future of DeepResearch
The paper concludes by reflecting on the future of DeepResearch, highlighting areas like the “art of query refinement,” where AI could proactively identify knowledge gaps and generate valuable clarification questions. It also calls for rethinking search evaluation for research-oriented tasks, moving beyond simple answer retrieval to metrics like evidence coverage and perspective diversity. The DEEPRESEARCH-REPORTEVAL framework provides a crucial tool for measuring progress and guiding the development of these systems as they evolve from mere information assistants into intelligent research partners. You can find the full research paper here: Understanding DeepResearch via Reports.
