TLDR: IMO-Bench is a new suite of benchmarks (IMO-AnswerBench, IMO-ProofBench, IMO-GradingBench) designed to rigorously evaluate AI models’ mathematical reasoning at the International Mathematical Olympiad level. It moves beyond short answers to assess proof-writing and grading capabilities, with the Gemini Deep Think model achieving gold-level performance. The benchmarks aim to foster development of robust, verifiable AI reasoning.
Advancements in artificial intelligence, particularly large language models, have shown impressive progress in mathematical reasoning. However, existing evaluation methods often fall short, either being too easy or focusing solely on correct short answers, which doesn’t truly assess a model’s deep reasoning capabilities. To address this, a new suite of benchmarks called IMO-Bench has been introduced, designed to evaluate AI models at the challenging level of the International Mathematical Olympiad (IMO).
IMO-Bench is a comprehensive suite that includes three distinct benchmarks. The first, IMO-AnswerBench, features 400 diverse Olympiad problems that require verifiable short answers. These problems have been carefully selected from past competitions and modified by experts to prevent memorization, ensuring models demonstrate genuine reasoning rather than recalling pre-seen solutions. The problems cover a wide range of topics including Algebra, Combinatorics, Geometry, and Number Theory, with varying difficulty levels from pre-IMO to IMO-Hard.
The second benchmark, IMO-ProofBench, takes evaluation to the next level by focusing on proof-writing capabilities. This benchmark consists of 60 problems, both basic and advanced, that demand models to generate complete and rigorous mathematical proofs. It includes detailed grading guidelines to facilitate consistent evaluation, moving beyond just getting the right answer to assessing the logical steps and coherence of an argument. The advanced set even includes novel problems crafted by IMO medalists, pushing the boundaries of AI reasoning.
Finally, IMO-GradingBench is introduced to evaluate a model’s ability to assess the quality of a given proof. This benchmark comprises 1000 human-graded solutions to problems from the advanced IMO-ProofBench, providing a valuable resource for developing and improving automated grading systems for long-form answers. This is crucial for scaling research in mathematical reasoning where human expert evaluation can be time-consuming and costly.
The development of IMO-Bench played a significant role in the historic achievement of the Gemini Deep Think model, which attained gold-level performance at IMO 2025. This model achieved an impressive 80.0% on IMO-AnswerBench and 65.7% on the advanced IMO-ProofBench, significantly outperforming other non-Gemini models. The research also highlights the effectiveness of automated graders built with Gemini reasoning, showing a strong correlation with human evaluations.
Also Read:
- Unveiling AI’s Discrete Math Challenges with CombiGraph-Vis
- GeoFM: A New Approach to Boost Geometric Reasoning in AI Models
The paper emphasizes that robust mathematical reasoning requires more than just correct answers; it demands verifiable, deep, and logical thought processes. By releasing IMO-Bench to the research community, the creators hope to encourage a shift in focus towards developing AI systems that can truly understand and generate complex mathematical arguments. You can find more details about this groundbreaking work at the official IMO-Bench website: IMO-Bench Research Paper.
