TLDR: NUMINA is a novel benchmark designed to evaluate multimodal large language models (MLLMs) on their ability to perform precise numerical reasoning and spatial understanding in 3D indoor environments. The benchmark, created using an automated pipeline, features diverse question types, including fact validation, prompt matching, and challenging numerical inference tasks like distance and volume estimation. Evaluations show that while current MLLMs perform well on non-numerical tasks, they significantly struggle with fine-grained 3D numerical reasoning, achieving very low accuracy in precise computations. This highlights a critical limitation in current AI architectures and the need for specialized geometric reasoning modules and enhanced 3D spatial supervision in training.
Recent advancements in artificial intelligence, particularly in multimodal large language models (MLLMs), have shown impressive capabilities in understanding and processing information from images and text. However, extending these capabilities to complex three-dimensional (3D) environments presents a unique set of challenges. While 2D vision-language tasks have seen significant improvements, AI models often struggle with the intricacies of spatial reasoning and precise numerical measurements in 3D spaces.
To address this critical gap, researchers have introduced a groundbreaking new benchmark called NUMINA. This benchmark, whose full details can be found in the research paper NUMINA: A Natural Understanding Benchmark for Multi-dimensional Intelligence and Numerical Reasoning Abilities, is the first of its kind designed to enhance multimodal indoor perceptual understanding by focusing on multi-dimensional intelligence and numerical reasoning abilities.
What is NUMINA?
NUMINA is a comprehensive dataset featuring 74,526 question-answer pairs, specifically tailored for fine-grained spatial understanding and numerical reasoning within indoor environments. Unlike previous benchmarks that offered only coarse, global annotations, NUMINA provides detailed, multi-scale annotations, including object center coordinates, bounding box dimensions, and convex hull distances between objects. This rich annotation allows for value-based inference and supports a diverse range of numerical reasoning question types.
The benchmark categorizes questions into three levels of increasing difficulty: Fact Validation (FV), which involves binary “yes” or “no” questions assessing factual accuracy; Prompt Matching (PM), consisting of multiple-choice questions that demand stronger spatial comprehension; and Numerical Inference (NI), the most challenging task, requiring precise numerical outputs for quantities, volumes, or distances.
How was NUMINA created?
The creation of NUMINA was streamlined through an automated annotation pipeline called NUMINA-Flow. This innovative system integrates large language model (LLM) rewriting and rule-based self-verification. It extracts Numerical Ground Truth (NGT) from the ScanNet dataset, then uses advanced LLMs like GPT-4o to generate diverse question templates. These templates are then validated through rule-based and manual verification processes to ensure clarity, accuracy, and a balanced data distribution. Additionally, non-numerical questions from the ScanQA dataset were rewritten and integrated to further enhance diversity.
Evaluating AI Performance
To assess the capabilities of current AI models, several state-of-the-art open-source LLMs, including Vicuna, Qwen, Phi, Mistral, and DeepSeek, were evaluated on the NUMINA benchmark. These models were integrated into the Chat-Scene framework, which allows for the simultaneous processing of 3D point clouds, 2D images, and textual inputs.
The results revealed significant challenges for existing LLMs in 3D numerical inference and spatial reasoning. While models performed well on non-numerical tasks (achieving over 75% accuracy in Prompt Matching and non-numerical Fact Validation), their accuracy dropped considerably in distance-related tasks (around 54%). The Numerical Inference category proved to be the most difficult, particularly for distance and volume estimation, where accuracy remained strikingly low, often below 3% even with a 5% error margin.
Why do AI models struggle?
The research paper highlights that current MLLMs struggle with fine-grained numerical reasoning in 3D settings primarily because their underlying Transformer architectures lack inherent inductive biases for geometry. These models often treat numbers as discrete tokens without a true understanding of magnitude, and they are not explicitly designed to encode complex spatial relationships like distances, angles, or volumes. Furthermore, standard pretraining data for LLMs typically lacks explicit 3D-spatial supervision or grounded numerical examples, leaving models unprepared for the geometric computations required by NUMINA.
This suggests that simply scaling up model size is not enough. Future advancements will require fundamental architectural changes, including specialized geometric reasoning modules and training paradigms that incorporate explicit 3D spatial supervision.
Also Read:
- EngiBench: A New Standard for Assessing AI in Engineering Challenges
- Assessing Multimodal AI: Daily Tasks Reveal Gaps in General Intelligence
The Path Forward
NUMINA serves as a crucial benchmark, highlighting the limitations of current AI models in precise 3D spatial and numerical reasoning. It underscores the need for further research and development in creating more robust and intelligent multimodal AI systems capable of truly understanding and interacting with our three-dimensional world. The dataset and source codes are publicly available for researchers to build upon and contribute to these vital advancements.
