TLDR: TIGeR is a framework that enhances Vision-Language Models (VLMs) for robotics by enabling them to perform precise geometric computations using external tools. It addresses the limitation of VLMs in achieving centimeter-level accuracy by allowing them to generate and execute computational code for tasks like pose estimation and trajectory generation. Supported by a new 300K-sample dataset (TIGeR-300K) and a two-stage training pipeline, TIGeR achieves state-of-the-art performance in geometric reasoning and demonstrates high precision in real-world robotic manipulation.
Vision-Language Models (VLMs) have made significant strides in understanding and interpreting the world around us, especially in spatial reasoning. They can tell us if an object is ‘left of’ another or ‘reachable’. However, when it comes to the precise, centimeter-level accuracy needed for real-world robotics, these models often fall short. They tend to treat geometric problems as simple pattern recognition tasks, overlooking crucial metric information from depth sensors and camera calibrations.
This limitation means that while VLMs can understand qualitative spatial relationships, they struggle with quantitative accuracy – the ability to compute exact 3D poses, rotation matrices, or collision-free trajectories. This gap prevents them from performing the exact manipulations and planning essential for robots operating in physical environments.
Introducing TIGeR: Tool-Integrated Geometric Reasoning
To bridge this gap, researchers have introduced TIGeR (Tool-Integrated Geometric Reasoning), a novel framework that transforms VLMs from mere perceptual estimators into powerful geometric computers. Instead of trying to embed complex geometric operations directly within neural networks, TIGeR empowers models to recognize when geometric reasoning is needed, synthesize appropriate computational code, and then invoke specialized external libraries for exact calculations.
This approach allows VLMs to leverage precise metric cues from depth sensors and camera calibration, enabling them to move beyond approximations to deliver the high accuracy required for robotic tasks.
How TIGeR Works: A Two-Pronged Approach
TIGeR operates through a hierarchical workflow, categorizing tools into two main groups:
-
Visual Perception Tools: These tools extract essential sensory information. Examples include retrieving camera intrinsic and extrinsic parameters, querying depth information from sensors, and generating precise object segmentation masks (using models like SAM2) from coarse VLM outputs like 2D bounding boxes.
-
Geometric Computation Tools: These tools process the data from visual perception to derive geometric properties. They can convert 2D bounding boxes into 3D bounding boxes, project 3D points onto a 2D image plane, and execute arbitrary computations (like inter-object distance calculations or pose estimations) through a code generation subroutine (using models like Qwen3-Coder). The generated code is run in a sandboxed environment, allowing for progressive reasoning and integration of intermediate results.
The TIGeR-300K Dataset and Training Pipeline
To train models effectively within this new paradigm, the researchers created TIGeR-300K, a comprehensive dataset of 300,000 high-quality samples. This dataset is specifically designed for tool invocation and geometric reasoning in embodied scenarios, covering tasks like point transformations, pose estimation, and spatial compatibility checks. Each sample includes problem statements, solutions, and the complete sequence of tool invocations and intermediate computations.
TIGeR is trained using a two-stage pipeline:
-
Supervised Fine-Tuning (SFT): This initial stage teaches the model the basic ability to generate reasoning chains and use tools correctly, leveraging the TIGeR-300K dataset.
-
Reinforcement Fine-Tuning (RFT): Following SFT, RFT refines the model’s generalized tool-integrated geometric reasoning capabilities. It uses a novel hierarchical reward design that evaluates tool use from both outcome-based (e.g., structural correctness, final answer accuracy) and process-based (e.g., correct tool selection, parameter accuracy, code execution success) perspectives.
Key Advantages for Robotics
TIGeR offers several significant advantages for robotics:
-
Accurate Spatial Localization: It enables precise prediction of 3D positions, keypoints, and trajectories by leveraging existing 3D foundation models, overcoming the limitations of data-driven VLMs in quantitative spatial reasoning.
-
Unified Reasoning Across Viewpoints: The method is viewpoint-invariant, supporting consistent numerical reasoning within a unified 3D coordinate system, even with uncalibrated multi-view cameras.
-
Interpretability and Adaptability: By explicitly invoking tools and exposing intermediate steps, TIGeR provides clear reasoning and allows for easy integration of new, state-of-the-art tools without costly retraining.
Also Read:
- Robots Learn to Grasp Better with Visual Chain-of-Thought
- Boosting Robot Precision with Internal Model Confidence
Real-World Impact and Performance
Extensive experiments demonstrate TIGeR’s superior performance. It achieves state-of-the-art results on geometric reasoning benchmarks and, crucially, delivers centimeter-level accuracy in real-world robotic manipulation tasks. For instance, in tasks requiring precise placement (e.g., “0.1m to the right of the peach”) or handling occlusions (e.g., placing an object “to the back of the brown toy”), TIGeR significantly outperforms baselines that lack depth-aware reasoning.
This framework represents a crucial step forward in enabling robots to perform complex, precise manipulations in dynamic, real-world environments. For more details, you can read the full research paper here.
