TLDR: ProfBench is a new benchmark with over 7000 expert-evaluated tasks across Physics, Chemistry, Finance, and Consulting, designed to assess large language models’ ability to handle complex, real-world professional documents and generate comprehensive reports. It introduces affordable and unbiased LLM-Judges and reveals that even state-of-the-art models struggle, with top performers achieving only 65.9% accuracy, highlighting significant challenges and disparities between proprietary and open-weight models. The benchmark emphasizes the importance of grounding documents and reasoning for high-quality professional responses.
Evaluating the true capabilities of large language models (LLMs) often feels like a game of Sudoku: verifying a solution is easy, but coming up with one can be incredibly difficult. For LLMs, this challenge has historically limited evaluations to straightforward tasks like math problems or short question-answering. However, real-world applications demand much more – LLMs need to process complex professional documents, synthesize information, and generate detailed reports.
A new benchmark called ProfBench aims to bridge this gap. Introduced by researchers from NVIDIA and other institutions, ProfBench is designed to push the boundaries of LLM evaluation beyond simple, verifiable answers. It focuses on tasks that require deep professional knowledge, similar to what a PhD or MBA holder would encounter in their field.
What is ProfBench?
ProfBench is a comprehensive dataset featuring over 7,000 pairs of responses and criteria, all meticulously evaluated by human experts. These experts hold professional qualifications such as Physics PhDs, Chemistry PhDs, Finance MBAs, and Consulting MBAs, ensuring that the tasks reflect genuine professional challenges. The benchmark covers 80 distinct tasks, equally divided among these four demanding domains.
Unlike many existing benchmarks that rely on simple multiple-choice or short-answer formats, ProfBench tasks are open-ended. They often involve multiple related sub-questions and require generating multi-page reports, mimicking real-world scenarios where professionals tackle complex problems. For instance, a Finance MBA task might ask an LLM to assess a new business unit’s potential within an investment bank, requiring detailed analysis of financial mechanisms like securitization and risk assessment. Similarly, a Physics PhD task could involve complex calculations related to axion models, while a Chemistry PhD task might delve into multi-step acid-base titration problems.
Building the Benchmark: Experts at Work
The creation of ProfBench was a rigorous process. Annotators with relevant PhDs, MBAs, or equivalent experience were recruited globally. They underwent tests to confirm their domain expertise and understanding of the annotation tasks. Each annotator was responsible for an entire task, from prompt ideation to rubric creation and response annotation, ensuring that the tasks captured their personal expertise. Importantly, LLMs were explicitly disallowed at any stage of the annotation process to maintain human-expert integrity.
For each task, annotators created between 15 to 60 criteria to score responses. These criteria are independently gradable and collectively cover all aspects of response quality, including logical validity, mathematical correctness, completeness of reasoning, and even stylistic elements like formatting and clarity. The majority of criteria (62.9%) focus on reasoning, while 34.1% assess information extraction, and a smaller portion (3.0%) addresses style.
Evaluating LLMs as Judges and Generators
ProfBench evaluates LLMs in two key roles: as ‘LLM-Judges’ that assess responses against the expert-created rubrics, and as ‘Report-Generators’ that produce the initial responses to the complex prompts.
For LLM-Judges, the benchmark aims to create robust and affordable evaluation tools. It addresses issues like ‘self-enhancement bias,’ where LLMs might unfairly favor responses from their own model family. By mitigating this bias and significantly reducing evaluation costs (by 2-3 orders of magnitude compared to existing rubric-based evaluations), ProfBench makes high-quality LLM assessment more accessible.
The findings for LLM-Judges showed that proprietary models like Google’s Gemini-2.5-Pro and OpenAI’s GPT-4.1 generally performed best. However, open-weight models like Kimi-K2-0711 and GPT-OSS-120B were not far behind, offering comparable performance at a fraction of the cost. The study also explored the impact of ‘thinking’ (enabling multi-step reasoning) in LLM-Judges, finding that it generally improves performance, particularly for complex Physics, Chemistry, and Style-related criteria.
When evaluating LLMs as Report-Generators, the benchmark revealed significant challenges even for state-of-the-art models. The top-performing model, GPT-5-high, achieved an overall performance of only 65.9%. This score highlights the benchmark’s difficulty, especially when compared to other popular benchmarks where GPT-5 achieves much higher scores. Among the domains, Physics proved most challenging (49.3%), followed by Finance (63.8%), Chemistry (70.6%), and Consulting (80.0%).
The research also identified notable performance disparities between proprietary and open-weight models, particularly in Finance, Chemistry, and Consulting. This suggests that open-weight models might have less in-domain training data for these areas compared to code or math-related tasks. Similar to LLM-Judges, enabling ‘thinking’ in report-generating LLMs generally led to performance improvements, though the gains varied.
The Importance of Grounding Documents
A crucial aspect of ProfBench is the use of ‘grounding documents’ – relevant public internet documents (like PDFs or CSVs) provided alongside the prompt. An ablation study demonstrated that removing these documents significantly reduced LLM performance (by 9.4% to 11.9%), especially for information extraction tasks. While adding web search capabilities recovered some performance, it also introduced a large amount of input tokens, indicating that precise retrieval remains an area for improvement. This emphasizes that providing explicit, relevant documents is vital for LLMs to generate high-quality, professionally accurate responses.
Also Read:
- Assessing AI Judges: A New Benchmark for Web Development Quality
- Assessing How Well Large Language Models Simulate Human Behavior with SIMBENCH
Looking Ahead
ProfBench represents a significant step forward in evaluating LLMs for complex, open-ended problems with real-world value. By providing expert-curated rubrics across diverse professional domains and making evaluation more affordable and unbiased, it aims to catalyze progress in LLM capabilities, particularly for open-weight models in less-explored areas like Chemistry, Consulting, and Finance. You can read the full research paper here: ProfBench: Multi-Domain Rubrics Requiring Professional Knowledge to Answer and Judge.
