TLDR: CombiGraph-Vis is a new 1,135-problem benchmark for discrete mathematical reasoning, including 35% image-based problems. It reveals significant performance gaps in AI models, especially with visual content and susceptibility to multiple-choice distractors, providing a detailed tool for advancing multimodal math reasoning research.
Researchers have introduced CombiGraph-Vis, a new benchmark designed to test the discrete mathematical reasoning abilities of artificial intelligence models. This benchmark is significant because it addresses a growing need for more challenging and diverse datasets in the field of AI, particularly in areas where current benchmarks are showing “ceiling effects,” meaning top models are already performing too well, making it hard to differentiate their capabilities.
CombiGraph-Vis features 1,135 problems spanning 13 different domains of discrete mathematics. These problems come in three formats: short-answer, multiple-choice, and yes/no. A notable aspect of this benchmark is its multimodal nature, with 35% of the problems including images. The structure within these images is crucial for solving the problems, highlighting a key area where AI models often struggle.
The creation of CombiGraph-Vis involved a meticulous process. Each problem is accompanied by a verified solution and technique labels, which categorize the mathematical methods required to solve them. The entire dataset was carefully curated and validated using agentic workflows, with human oversight to ensure high consistency and fidelity. This rigorous validation process helps ensure the benchmark is reliable for evaluating AI performance.
Initial evaluations of various AI model families on CombiGraph-Vis have revealed a wide range of performance, with accuracy scores varying from 16% to 78%. A particularly interesting finding is the sharp drop in performance observed on problems that include images. This suggests that while AI models are advancing in text-based reasoning, interpreting and reasoning with structured visual information, such as graphs, grids, and geometric diagrams, remains a significant challenge.
For standalone multiple-choice problems, the evaluations also uncovered a clear gap between the accuracy of selecting the correct answer and the accuracy of choosing an answer that is merely among the provided options. This indicates that models are susceptible to “trap choices” – distractors deliberately designed to mislead, a common feature in competitive math problems. This vulnerability suggests that models often rely on pattern recognition rather than robust derivation of solutions.
The benchmark places a strong emphasis on reasoning over combinatorial objects like graphs and grids. The detailed technique labels allow researchers to conduct fine-grained analysis, identifying specific mathematical approaches where models excel or falter. This level of detail is invaluable for guiding future research efforts in building more robust and capable AI systems for mathematical reasoning.
Also Read:
- GeoFM: A New Approach to Boost Geometric Reasoning in AI Models
- SIGMA: Enhancing AI’s Mathematical Reasoning with Collaborative Agents and Smart Search
The researchers have made the dataset, solutions, technique labels, and evaluation code publicly available on GitHub, encouraging further research and development in this critical area. You can find more details about this work in the full research paper: CombiGraph-Vis: A Curated Multimodal Olympiad Benchmark for Discrete Mathematical Reasoning.
