TLDR: MaRVL-QA is a new benchmark that evaluates Multimodal Large Language Models (MLLMs) on deep mathematical and spatial reasoning directly from images, using semantically sparse mathematical surface plots. It features two tasks: Topological Counting (identifying features like local maxima) and Transformation Recognition (identifying geometric changes). Evaluations show that even state-of-the-art MLLMs struggle significantly, often relying on superficial heuristics and failing to scale with complexity, highlighting a critical gap in their reasoning capabilities.
Multimodal Large Language Models (MLLMs) have shown impressive capabilities in understanding and describing the visual world, but a new benchmark, MaRVL-QA, reveals a significant gap in their ability to perform deep mathematical and spatial reasoning directly from images.
Introducing MaRVL-QA: A New Frontier for MLLM Evaluation
Developed by researchers from Waymo and Google, MaRVL-QA (Mathematical Reasoning over Visual Landscapes) is designed to rigorously test these core reasoning skills. Unlike natural images which can introduce semantic “noise,” MaRVL-QA uses mathematical surface plots to isolate the task of reasoning, providing a clear and unambiguous testbed.
The benchmark introduces two novel tasks:
Topological Counting: This task challenges MLLMs to identify and accurately count specific features on a mathematical surface, such as local maxima (peaks) or local minima (valleys).
Transformation Recognition: Here, models are presented with an original plot and a transformed version, and must identify the geometric transformation applied, such as rotations or translations.
The creation of MaRVL-QA involved a sophisticated pipeline. It starts with a curated library of diverse mathematical functions, from which over 80,000 question-answering pairs are programmatically generated. A crucial step is rigorous, multi-stage filtering to remove any perceptual ambiguities, ensuring that every question has a single, objective correct answer.
Why MaRVL-QA is Different
Existing benchmarks for visual question answering often focus on data extraction from charts or reasoning about discrete objects in synthetic scenes. Mathematical reasoning benchmarks for language models typically rely on text-based problems. MaRVL-QA bridges a critical gap by requiring models to comprehend mathematical concepts directly from visual data, interpreting the topological and geometric features of a visualized surface.
To prevent models from relying on superficial heuristics like text extraction or axis label changes, MaRVL-QA employs specific rendering strategies. For instance, in transformation tasks, both original and transformed plots are rendered within the same expanded domain with identical axis labels, forcing models to recognize shape reorientation or movement within a static frame.
Key Findings: MLLMs Struggle
Extensive evaluations on MaRVL-QA, including a high-quality 2,748-item test set called MaRVL-QA-Mini, revealed that even state-of-the-art MLLMs struggle significantly. Models often resort to superficial heuristics rather than robust spatial reasoning.
In the Topological Counting task, the highest-performing model achieved only 58.91% accuracy. A notable finding was that models consistently performed better at counting maxima (bright peaks) than minima (dark valleys), suggesting a bias towards visual salience. More critically, accuracy sharply declined as the number of features to be counted increased, indicating a failure in algorithmic counting beyond small quantities (subitizing).
For Transformation Recognition, while top models showed proficiency in translation tasks (over 78% accuracy), rotation tasks proved more challenging for all, with performance capped around 50-54%. The analysis also exposed inconsistencies within model families, with some exhibiting erratic performance or biases towards specific transformation types, suggesting the learning of narrow, heuristic-based strategies rather than generalized spatial understanding.
Failure analysis further highlighted two common patterns: defaulting to a “No Change” option when uncertain, or rigidly adhering to a single preferred (and often incorrect) option, indicating a breakdown in genuine reasoning.
Also Read:
- Keeping Pace with AI: A Live Benchmark for Scientific Understanding
- Addressing Age Bias in Medical AI: Introducing the PediatricsMQA Benchmark
Looking Ahead
MaRVL-QA provides a challenging new tool for the research community to measure progress, expose model limitations, and guide the development of MLLMs with more profound reasoning abilities. Future work will focus on developing new model architectures and training paradigms to improve systematic, procedural reasoning, and extending the benchmark with even more complex mathematical concepts.
For more details, you can read the full research paper here: MaRVL-QA: A Benchmark for Mathematical Reasoning over Visual Landscapes.
