TLDR: SpatialVTS is a new method that significantly enhances the spatial reasoning abilities of Vision Language Models (VLMs). It achieves this through two phases: Spatial Visual Thinking, which identifies both obvious and potential visual cues in an image, and Spatial Textual Thinking, which performs step-by-step logical reasoning based on these cues. The method also involves extensive manual correction and restructuring of training datasets to improve data quality and incorporate detailed reasoning processes. Experiments show SpatialVTS outperforms other models in spatial understanding tasks without needing additional visual information like depth maps.
Vision Language Models (VLMs) have made incredible strides in understanding images and text, but they often struggle with a crucial human ability: spatial reasoning. This involves understanding relationships in 2D and 3D space, like determining if one object is “above” another or measuring the distance between them. This limitation impacts critical applications such as visual question answering and robotics.
A new research paper titled “Enhancing Spatial Reasoning through Visual and Textual Thinking” introduces an innovative method called SpatialVTS (Spatial reasoning through Visual and Textual thinking Simultaneously) to tackle this challenge. Developed by a team of researchers including Xun Liang, Xin Guo, Zhongming Jin, Weihang Pan, Penghui Shang, Deng Cai, Binbin Lin, and Jieping Ye, this approach aims to significantly improve how VLMs perceive and reason about spatial relationships.
The core of SpatialVTS lies in its two-phase thinking process. The first phase is Spatial Visual Thinking. Unlike previous methods that might only focus on objects explicitly mentioned in a question, SpatialVTS is designed to automatically identify not just the obvious targets, but also “potential” objects in an image that could be crucial for spatial reasoning. For example, if you need to measure the vertical distance between a spire and a car, the model might identify a nearby building and use its estimated story height as a reference scale. This phase trains the model to generate specific tokens representing the approximate locations of these important visual cues.
Following this, the model enters the Spatial Textual Thinking phase. Here, the VLM takes the identified visual cues, along with the original question, and engages in a “long-term thinking” process. Instead of providing a direct, simple answer, the model is encouraged to logically infer the solution step-by-step. This detailed reasoning process helps the model build connections between various visual targets and ultimately arrive at the correct answer. This approach is inspired by how humans often think through problems, breaking them down into smaller, logical steps.
A significant part of this research involved meticulously improving the datasets used for training. The researchers found that existing spatial reasoning datasets often had limitations, such as rigid input formats that required explicit masks or bounding boxes, poor data quality with incorrect answers, and a lack of detailed reasoning processes. To address this, the team manually corrected numerous errors, restructured the data to be more generalized (using plain text and image inputs with region labels), and added rich rationales that explain the logical steps to reach an answer. This “seeking the cause by grasping the result” strategy helped generate high-quality reasoning paths for training.
The experimental results for SpatialVTS are very promising. The model demonstrated substantial improvements in overall average performance across various spatial understanding tasks, including qualitative benchmarks that assess concepts like “above/below” and quantitative benchmarks that measure precise distances. Remarkably, SpatialVTS achieved these results without needing additional information like depth maps or segmentation masks, which some other advanced models rely on. Its performance was competitive even with models that do use such extra data, highlighting the effectiveness of its visual and textual thinking approach.
Also Read:
- TransPrune: Boosting Efficiency in Large Vision-Language Models Through Token Transition Analysis
- Bridging the Data Gap for Spatial AI with 2D-to-3D Conversion
This work represents a significant step forward in making Vision Language Models more capable of complex spatial reasoning, opening new possibilities for applications in areas like embodied intelligent systems and human-computer interaction. You can read the full research paper for more details. Read the full research paper here.
