TLDR: CritPt is a new benchmark created by over 50 physicists to test large language models (LLMs) on complex, unpublished, research-level physics problems. It includes 71 full challenges and 190 simpler checkpoints across various physics fields. Initial evaluations show that even state-of-the-art LLMs like GPT-5 struggle significantly with these problems, achieving very low accuracy on full challenges (4.0%) and only modest improvements with tools. This highlights a substantial gap between current AI capabilities and the demands of real-world physics research, providing a foundation to guide future AI development for scientific discovery.
Large language models, or LLMs, have made impressive strides in recent years, demonstrating remarkable abilities in areas like high-school math competitions and coding. However, a crucial question remains: can these advanced AI systems truly reason through the complex, open-ended challenges encountered in cutting-edge physics research? And what specific tasks do physicists actually want AI to help them with?
To answer these questions, a groundbreaking new benchmark called CritPt (Complex Research using Integrated Thinking – Physics Test) has been introduced. This is the first benchmark specifically designed to evaluate LLMs on unpublished, research-level reasoning tasks that span a wide array of modern physics disciplines. These include condensed matter, quantum physics, astrophysics, high energy physics, mathematical physics, and many more.
The CritPt benchmark is a monumental effort, consisting of 71 comprehensive research challenges. These challenges are crafted to simulate full-scale research projects at an entry level, similar to what a junior graduate student might tackle. To provide more detailed insights, these challenges are further broken down into 190 simpler ‘checkpoint’ tasks. All of these problems are brand new, created by a collaborative team of over 50 active physics researchers, each drawing from their own ongoing work.
A key design principle behind CritPt is to ensure that the problems are ‘guess-resistant’ and ‘machine-verifiable’. This means that simple memorization or retrieval from existing data won’t work. The answers often involve complex numerical values or intricate symbolic expressions, requiring genuine reasoning to solve. The evaluation process is also highly sophisticated, using an automated grading system customized for advanced physics output formats, including support for numerical values, symbolic expressions, and executable Python functions with test cases.
The initial findings from evaluating current state-of-the-art LLMs on CritPt reveal a significant gap between AI capabilities and the demands of realistic physics research. For the full, complex challenges, the best average accuracy achieved by a base model was a mere 4.0%, by GPT-5 (high). Even when equipped with coding tools, this accuracy only moderately rose to about 10%. This suggests that while LLMs show some early promise on isolated, simpler tasks (checkpoints), they are still far from reliably solving full-scale research problems.
Interestingly, adding web search capabilities to the LLMs provided only a marginal increase in performance, confirming that CritPt’s design effectively prevents models from simply looking up answers. The research also highlighted a concern regarding the reliability of LLM outputs. When a stricter metric was applied – requiring at least four out of five independent runs to yield a correct answer – the performance dropped sharply across all models, indicating high variability in their reasoning processes. This means that even when an LLM gets an answer right, it might not do so consistently, which is a critical issue for high-stakes scientific applications.
Also Read:
- Advancing Scientific Verification for Language Models with SCI-Verifier
- Beyond a Single Roll: Why Repetitions Are Key to Reliable LLM Evaluations
The creators of CritPt hope that this benchmark will serve as a crucial tool to guide the development of scientifically grounded AI tools. By providing a realistic yet standardized evaluation framework, it clearly identifies the areas where current AI models fall short and offers a foundation for building more capable and reliable reasoning models for scientific discovery. For more detailed information, you can refer to the original research paper: Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmark.
