TLDR: DeepPHY is a new benchmark framework that evaluates Vision Language Models (VLMs) on their ability to understand and reason about physical principles in interactive, simulated environments. It includes six diverse physics-based games and puzzles. The research found that even advanced VLMs struggle significantly with complex physical interactions, long-term planning, and dynamic adaptation, often failing to translate their descriptive knowledge of physics into precise, predictive control actions.
Vision Language Models, or VLMs, have shown impressive capabilities in understanding static images and text. However, when it comes to navigating and interacting with dynamic, real-world environments that involve complex physics, these models often fall short. Tasks that require precise action planning, advanced spatial reasoning, and continuous strategy refinement, like playing a game of billiards or solving a physics puzzle, prove to be particularly challenging for current AI.
To address this gap, a new benchmark framework called DeepPHY has been introduced. DeepPHY is designed to systematically evaluate how well VLMs understand and reason about fundamental physical principles. It does this by immersing AI agents in a series of challenging simulated environments, moving beyond simple question-answering formats to test interactive physical reasoning.
What is DeepPHY?
DeepPHY integrates six diverse and challenging physics-based simulation environments, none of which have been previously combined for benchmarking agentic VLMs. These environments include:
- PHYRE: 2D puzzles where agents place objects to trigger chain reactions.
- I-PHYRE: Interactive physics scenarios requiring precise timing to remove obstacles.
- Kinetix: A 2D physics platform generating control tasks like robotic locomotion.
- Pooltool: A high-fidelity billiards simulation.
- Angry Birds: The popular game where birds are launched to dismantle structures and eliminate pigs.
- Cut the Rope: A puzzle game where agents cut ropes and use props to guide candy.
Unlike traditional benchmarks that might test physical reasoning through static questions or text-based problems, DeepPHY puts agents directly into interactive sandboxes. Success in these environments depends on performing actions and understanding their physical consequences over time.
How Does DeepPHY Work?
The researchers behind DeepPHY have standardized the observation and action spaces across these diverse environments to make them more accessible for VLMs. For instance, continuous actions like placing a ball at any coordinate are converted into discrete selections from a grid. Visual scenes are often augmented with grids or numerical IDs to help models identify interactive objects, shifting the challenge from basic object detection to understanding physical dynamics and planning manipulations.
The evaluation protocol categorizes planning strategies into ‘in-advance planning’ (where a complete solution is devised upfront) and ‘on-the-fly planning’ (sequential, turn-by-turn interaction). They also tested two prompting strategies: Vision-Language-Action (VLA), where the model directly outputs an action, and World Model (WM), which also requires the model to predict environmental changes resulting from its action.
Also Read:
- VL-DAC: Training Vision-Language Models for Real-World Action Through Synthetic Environments
- OmniPlay: A New Benchmark Reveals Strengths and Weaknesses in Omni-Modal AI Reasoning
Key Findings: AI’s Struggle with Physics
The extensive evaluation across the DeepPHY suite revealed significant limitations in current VLMs:
- Overall Performance: Most models, especially open-source ones, struggle to surpass even random action baselines. This indicates a lack of deep understanding of underlying physical principles and zero-shot planning ability.
- State-of-the-Art Limitations: Even leading closed-source models like GPT-o3, Gemini-2.5-Pro, and Claude 4.0 Opus, while performing better than others, still show a stark performance gap compared to humans. Their success rates are considerably lower than desired.
- Learning from Failure: In environments like PHYRE, models show slow improvement even after multiple failed attempts, suggesting they struggle to learn effectively from feedback and revise their strategies.
- The World Model Disconnect: A counter-intuitive finding was that the World Model (WM) prompting strategy often failed to improve, and sometimes even degraded, performance compared to the simpler VLA approach. This suggests that even if models can describe a potential physical outcome, this descriptive knowledge doesn’t necessarily translate into improved procedural control or an accurate predictive internal world model.
- Brute-Force vs. Reasoning: In games like Pooltool, some models achieved high success rates not through nuanced physical reasoning (like controlling cue ball spin), but by consistently applying a simple, brute-force heuristic (e.g., maximum power shots). This highlights a lack of true strategic understanding.
- Complex Dynamics: In games like Angry Birds and Cut the Rope, models struggled immensely with multi-stage physics tasks requiring precise timing and understanding of chain reactions. Their failures often stemmed from incorrect timing or sequencing, demonstrating fundamental limitations in spatiotemporal reasoning for dynamic physical processes.
The research concludes that there is a fundamental disconnect between a VLM’s ability to describe physical phenomena and its ability to use that knowledge to predict and control outcomes in dynamic environments. DeepPHY serves as a rigorous testbed to benchmark these limitations and facilitate the development of more physically grounded AI agents. You can find the full research paper at https://arxiv.org/pdf/2508.05405.
