TLDR: CapGeo is a new framework that significantly improves how Multimodal Large Language Models (MLLMs) solve geometry problems. It works by converting complex geometric diagrams into simple, structured textual captions, which helps MLLMs overcome their struggles with visual perception. The research also introduces CapGeo-Bench, a dataset and evaluation method to assess the quality of these geometric captions, showing that better captions lead to better reasoning performance.
Multimodal Large Language Models (MLLMs) have shown incredible progress in understanding and generating human-like text, even excelling in complex textual reasoning tasks like the International Mathematical Olympiad. However, when these advanced AI models encounter geometry problems that involve visual diagrams, they often struggle. This significant gap suggests that the main hurdle isn’t their reasoning ability itself, but rather their difficulty in accurately interpreting geometric figures.
Recognizing this challenge, researchers have introduced CapGeo, a novel caption-assisted reasoning framework designed to bridge the gap between visual and textual understanding in MLLMs. The core idea behind CapGeo is simple yet powerful: since geometric figures can often be precisely described in concise textual form, converting the visual content of a diagram into a caption can significantly enhance an MLLM’s ability to solve geometry problems.
CapGeo works by first taking a geometric figure and generating a structured caption that describes its elements and relationships. This caption, along with the original problem statement, is then fed to the MLLM for reasoning. By providing a clear, text-based representation of the visual information, CapGeo helps the model bypass the complexities and potential ambiguities of direct visual perception, allowing it to leverage its strong textual reasoning capabilities more effectively.
The results of implementing CapGeo have been remarkable. For instance, the Qwen2.5-VL-72B model saw its performance on geometry tasks improve dramatically from a mere 8.6% (when relying solely on vision) to an impressive 59.0% with caption assistance. Similarly, Claude-Opus-4’s accuracy rose from 44.8% to 73.0%. These substantial gains underscore the framework’s effectiveness in addressing the visual understanding bottleneck in geometric reasoning.
Evaluating Geometric Captioning
To systematically evaluate and identify high-quality geometric captioning models, the researchers also developed CapGeo-Bench. This comprehensive dataset comprises 4,641 carefully curated figure-caption pairs, covering a wide range of geometric problems with varying difficulty levels and types, including Plane Geometry, Analytic Geometry, and Solid Geometry. The creation of CapGeo-Bench involved meticulous data collection from K-12 textbooks and rigorous manual annotation by experts with STEM backgrounds.
A crucial aspect of CapGeo-Bench is its innovative keypoint-based evaluation metric. Unlike general image captioning metrics, this method specifically assesses the quality of geometric captions across three dimensions: elements (identifying shapes, lines, points), spatial relations (describing relationships like parallel, perpendicular, intersection), and numerical relations (extracting values like lengths and angles). This fine-grained evaluation has been validated by mathematics experts and shows a strong correlation with how well an MLLM performs on downstream reasoning tasks when assisted by these captions. This means CapGeo-Bench can effectively guide the development and selection of superior captioning models.
Also Read:
- Unlocking LLM Reasoning Power with Just 90 Examples: A New Data Augmentation Method
- Seeing Inside LLMs: How Computational Graphs Reveal Reasoning Flaws
Remaining Challenges and Future Directions
Despite the significant advancements, the research highlights that geometric captioning still presents challenges. MLLMs currently demonstrate the weakest capability in the numerical dimension, often struggling to accurately match numerical values with their corresponding geometric elements. Performance also consistently drops as the difficulty level of the problems increases, particularly in Plane Geometry, which involves highly abstract and symbolic visual content.
The CapGeo framework and CapGeo-Bench benchmark collectively establish a new pathway for advancing geometric reasoning in MLLMs. By focusing on transforming visual diagrams into precise textual descriptions, this work provides a robust foundation for future research aimed at bridging the gap between visual and textual modalities, ultimately leading to more capable and reliable AI systems for complex mathematical problems. You can read more about this research in the full paper available here.
