TLDR: NEWTONBENCH is a new benchmark for evaluating how well LLMs can discover scientific laws. It uses “metaphysical shifts” to create novel, yet scientifically relevant, laws across 12 physics domains, making tasks scalable and resistant to memorization. Unlike previous benchmarks, it requires LLMs to actively experiment within complex simulated environments. Evaluations of 11 LLMs show that while frontier models have discovery capabilities, they are fragile, degrading with increased complexity and noise. Paradoxically, code assistance can hinder stronger models by promoting premature exploitation over broad exploration.
Large language models (LLMs) are rapidly advancing, showing impressive capabilities in areas like mathematical and logical reasoning, and agentic planning. These skills are crucial for scientific inquiry, leading to exciting questions about whether LLMs could independently discover scientific laws, much like Isaac Newton did centuries ago.
However, truly evaluating these capabilities has been a significant challenge. Existing benchmarks for scientific law discovery in LLMs face a fundamental problem: they struggle to balance scientific relevance, scalability, and resistance to memorization. Many current tests either use complex, obscure laws that are hard to scale, or they transform known laws in ways that LLMs might simply recall the original rather than genuinely reason. Others use synthetically generated laws, which are scalable and memorization-resistant, but lack real-world scientific relevance.
Furthermore, these benchmarks often oversimplify the scientific discovery process. They treat it as a static task of fitting a mathematical function to a dataset. In reality, scientific discovery involves actively exploring complex systems, designing experiments, and interpreting feedback to uncover hidden principles. Think of J.J. Thomson inferring the electron’s charge-to-mass ratio by manipulating cathode rays – a dynamic, interactive process.
Introducing NEWTONBENCH: A New Approach to AI Scientific Discovery
To address these critical limitations, researchers have introduced NEWTONBENCH, a novel benchmark designed to rigorously test the generalizable scientific law discovery abilities of LLM agents. This benchmark features 324 scientific law discovery tasks spanning 12 diverse physics domains.
NEWTONBENCH tackles the methodological trilemma by employing “metaphysical shifts.” This technique systematically alters the mathematical structure of canonical physical laws – for example, by changing operators or exponents. This generates a vast suite of laws that are conceptually grounded in physics but are physically novel. This approach ensures scalability, maintains scientific relevance, and prevents memorization, forcing LLMs to reason from first principles rather than relying on recall.
Beyond static function fitting, NEWTONBENCH elevates the evaluation to “interactive model discovery.” In this environment, an LLM agent must actively design experiments by specifying input parameters (like setting the mass of an object or initial velocity in a simulation) and then interpret the resulting feedback from the virtual environment. Crucially, the target law is embedded within a complex model that includes confounding variables, requiring the agent to disentangle these factors to uncover the hidden principles.
The benchmark also allows for fine-grained control over difficulty, with two independent dimensions: the intrinsic complexity of the target law (adjusted via metaphysical shifts) and the extrinsic complexity of the surrounding experimental system. This helps in pinpointing the breaking points of LLM capabilities.
Also Read:
- RECODE-H: A New Benchmark for Interactive Research Code Generation with Human Feedback
- Expert-Built Benchmark Challenges AI in Frontier Physics Research
Key Findings: Fragile Capabilities and the Paradox of Tool Assistance
An extensive evaluation of 11 state-of-the-art LLMs on NEWTONBENCH revealed some fascinating insights. Frontier models like GPT-5 and Gemini-2.5-pro showed a clear, but often fragile, capability for discovery. Their performance significantly degraded as system complexity increased or when observational noise was introduced. This highlights that robust, generalizable discovery in complex, interactive environments remains a core challenge for the future of automated science.
Interestingly, the study uncovered a “paradoxical effect of tool assistance.” Providing a code interpreter, while boosting weaker models by offloading computation, actually hindered more capable models. This was attributed to a “premature shift from exploration to exploitation,” where advanced agents over-relied on the tool for local optimization, causing them to settle for suboptimal solutions instead of thoroughly exploring to find the globally correct law.
NEWTONBENCH offers a crucial tool for measuring true progress and guiding the development of next-generation AI agents capable of genuine scientific discovery. It provides a scalable, robust, and scientifically authentic testbed for understanding the limits and potential of LLMs in automated science. You can find more details about this research paper here: NewtonBench: Benchmarking Generalizable Scientific Law Discovery in LLM Agents.
