TLDR: A new benchmark called HIPHO evaluates 30 AI models on 13 high school physics Olympiad exams from 2024-2025, comparing their performance directly with human contestants. Closed-source MLLMs achieve gold medals but still lag behind top humans, while open-source models mostly reach bronze or silver levels. Diagram-based problems and optics are major challenges for AI, highlighting the need for improved multimodal, generative, and embodied reasoning for human-level physics mastery.
A new study titled “HIPHO: How Far Are (M)LLMs From Humans in the Latest High School Physics Olympiad Benchmark?” introduces a groundbreaking benchmark to evaluate the physics reasoning capabilities of large language models (LLMs) and multimodal large language models (MLLMs) against human performance in high school physics Olympiads. This research, conducted by a team of experts including Fangchen Yu, Haiyuan Wan, and Qianjia Cheng, addresses significant gaps in existing physics benchmarks by offering up-to-date, comprehensive, and human-aligned evaluations.
The HIPHO benchmark is the first of its kind to focus specifically on high school physics Olympiads, providing a direct comparison between AI models and human contestants. It features three core innovations designed to offer a more rigorous and realistic assessment. Firstly, it includes a comprehensive dataset compiled from 13 of the latest Olympiad exams from 2024–2025. These exams cover both international and regional competitions and feature mixed modalities, ranging from text-only problems to those requiring complex diagrams.
Secondly, HIPHO employs a professional evaluation method that aligns closely with human examiners. It uses official marking schemes to grade solutions at both the answer and step levels, ensuring a fine-grained and domain-specific assessment. This approach allows for partial credit, reflecting the nuanced scoring in real-world physics competitions.
Thirdly, and perhaps most notably, HIPHO enables direct comparison with human contestants. Models are awarded gold, silver, and bronze medals based on official medal thresholds, providing a clear and intuitive way to understand how AI performance stacks up against top human students.
The large-scale evaluation involved 30 state-of-the-art (M)LLMs. The findings reveal a clear performance hierarchy. Closed-source reasoning MLLMs, such as Gemini-2.5-Pro and GPT-5, demonstrated strong capabilities, achieving between 6 and 12 gold medals across the 13 Olympiads. However, even these top models still showed a significant gap when compared to the very best human contestants, particularly in highly challenging exams like the International Physics Olympiad (IPhO) and the European Physics Olympiad (EuPhO).
Open-source MLLMs generally performed at or below the bronze level, with Intern-S1 being a notable exception, securing four gold medals. Open-source LLMs, surprisingly, showed promising progress, occasionally earning gold medals, especially in easier contests like F=MA. Despite these advancements, they still lagged considerably behind the top human students.
The study also delved into specific challenges for MLLMs. Diagram-based problems consistently proved more difficult than text-only ones, with scores declining as visual complexity increased. Problems requiring the interpretation of variable-based graphs and the extraction of quantitative information from data figures presented particular hurdles. Optics was identified as the most challenging physics field for all models, likely due to its reliance on both diagram interpretation and precise symbolic derivations.
The researchers highlight that for AI to achieve true human-level physics reasoning, advancements are needed in three key areas: multimodality for robust integration of textual and visual inputs, generative ability to produce diagrams and functional plots, and embodied ability to support experimental reasoning and physical interaction. Without progress in these dimensions, even the most capable (M)LLMs will remain limited compared to the holistic problem-solving skills of human contestants.
Also Read:
- Assessing AI’s Financial Acumen: A Deep Dive into Language Models and CFA Exams
- AI’s Unequal Advice: Examining Bias in Educational LLMs
The HIPHO benchmark is open-source and available for further research and development, aiming to drive progress in multimodal physical reasoning. You can find more details about this research paper at this link.
