TLDR: PHYS GYM is a new benchmark and simulation platform designed to evaluate how large language models (LLMs) perform scientific discovery in interactive physics environments. It uniquely controls the amount of prior knowledge given to the LLM, allowing researchers to see how models adapt to varying information levels, from full context to completely anonymized variables. Experiments show that LLMs generally perform better with more prior knowledge, especially on complex tasks, and adapt their exploration strategies based on the information available.
The field of artificial intelligence is rapidly advancing, with Large Language Models (LLMs) showing remarkable capabilities across various domains. A particularly exciting area is their potential to accelerate scientific discovery. However, evaluating how well these AI agents truly reason and adapt in complex scientific environments, especially when faced with different levels of prior knowledge, has been a significant challenge.
To address this crucial gap, researchers have introduced PHYS GYM, a novel benchmark suite and simulation platform. This innovative tool is designed to rigorously assess the scientific reasoning abilities of LLM-based agents within interactive physics environments. Its core strength lies in its sophisticated control over the amount of prior knowledge provided to the AI agent, allowing for a detailed analysis of performance based on problem complexity and available information.
What is PHYS GYM?
PHYS GYM is more than just a dataset; it’s an interactive environment where AI agents can actively engage in physics discovery. Imagine a virtual lab where an AI can conduct experiments, gather data, and formulate hypotheses about underlying physical laws. This platform comprises a suite of interactive simulations, abstracting real-world physics problems into controllable settings.
How Does It Work?
At the heart of PHYS GYM is the concept of ‘controllable priors’. This means researchers can precisely adjust the information given to the AI. The platform offers four distinct levels of prior knowledge:
- Level 1: The AI receives full contextual information, including a detailed problem description and clear, meaningful names and descriptions for all variables (e.g., ‘mass’, ‘gravity’). At this level, solutions can often be deduced directly.
- Level 2: The textual problem description is removed, forcing the AI to rely more on variable descriptions and experimental data.
- Level 3: Variable descriptions are also removed, leaving only the meaningful variable names.
- Level 4: This is the most challenging level, where all explicit prior knowledge is withheld. Variable names are anonymized (e.g., ‘var1’, ‘var2’), turning the task into an interactive equation discovery problem where the AI must learn primarily through experimentation.
In each scenario, the AI agent iteratively designs experiments by proposing values for controllable variables and observing the outcomes. It then uses this feedback to refine its hypotheses about the underlying physical equation. PHYS GYM also enforces realistic constraints, such as a limited experimental budget, mirroring real scientific practice.
Evaluating Performance
The primary metric for success in PHYS GYM is the ‘Success Rate’, which measures if the AI’s proposed equation is mathematically equivalent to the true underlying physics equation. Beyond this, the platform uses auxiliary metrics to assess how consistently the AI’s hypotheses fit observed data (e.g., R2, MSE) and to quantify task difficulty based on equation length and variable count.
Key Findings from Experiments
The researchers tested several leading LLMs, including Gemini-2.5-flash, OpenAI-o4-mini, and Claude-3.7-Sonnet. The results provided valuable insights:
- Impact of Prior Knowledge: As expected, reducing prior knowledge generally led to a decrease in success rates for all models. ‘Thinking’ models like Gemini and OpenAI showed a substantial drop, indicating their reliance on contextual information. Interestingly, Claude-3.7-Sonnet showed less benefit from richer information at higher levels, suggesting a limited ability to leverage complex priors for reasoning.
- Prior Importance for Difficult Tasks: The study found that prior knowledge becomes even more critical for solving more complex problems (those with a higher number of variables). For tasks with 10 or more variables, models often only succeeded at the highest prior knowledge level.
- Hypothesis Adjustment: Gemini models demonstrated a strong ability to dynamically adjust their hypotheses in response to experimental outcomes, especially under conditions of greater uncertainty.
- Exploration Dynamics: With less prior knowledge, all models increased their number of interaction experiments, showing a greater reliance on active exploration. This highlights how models shift from pattern recognition to more exploratory behaviors when explicit information is scarce.
Also Read:
- Assessing Physics Understanding in AI Video Generation
- LLMs Learn to Think Smarter with Hierarchical Budget Policy Optimization
Conclusion
PHYS GYM represents a significant step forward in evaluating AI’s scientific discovery capabilities. By providing a controlled, interactive environment with adjustable prior knowledge, it allows researchers to understand not just whether an AI can solve a physics problem, but how it approaches scientific reasoning and adapts its strategies under varying informational landscapes. The findings underscore the importance of developing AI agents that can robustly reason and adaptively explore, moving beyond static benchmarks to truly become effective scientific partners. You can read the full research paper at arXiv.
