TLDR: Manual2Skill++ is a new vision-language framework that enables robots to perform complex assembly tasks by extracting detailed connection information directly from instruction manuals. It uses Vision-Language Models (VLMs) to parse symbolic diagrams and annotations, creating hierarchical graphs that explicitly model connector types, specifications, and placement locations. This approach allows for millimeter-level accurate part pose alignment through geometric constraint satisfaction, significantly improving robotic assembly precision. The framework was validated in simulation and real-world setups across diverse tasks like assembling IKEA furniture and toy models, demonstrating its effectiveness in understanding and executing connector-aware robotic assembly operations.
Robotic assembly, the process of teaching robots to put things together, has long faced a significant hurdle: the intricate details of how parts actually connect. While robots can be programmed to plan assembly sequences and position parts, the crucial ‘last mile’ of forming physical connections – whether it’s screwing, doweling, or fitting a mortise-tenon joint – often gets overlooked. This oversight limits their ability to handle real-world assembly tasks, where the precise selection and application of connectors are essential for success.
A new vision-language framework called Manual2Skill++ aims to change this by treating connections as a primary focus in robotic assembly. Inspired by how humans learn from step-by-step instruction manuals, this system automatically extracts detailed connection information, including connector types, specifications, quantities, and exact placement locations, directly from these manuals.
Understanding Assembly Through Hierarchical Graphs
Manual2Skill++ represents assembly tasks as hierarchical graphs. Imagine a family tree for an assembled product: leaf nodes are individual parts, intermediate nodes are sub-assemblies (like a partially built chair), and the root node is the final product. What makes this approach unique is the explicit modeling of connection relationships between components using ‘connection edges’. These edges carry vital attributes, such as the type of connection (e.g., mortise-tenon, dowel, screw), the specific connector used, and the precise attachment points on each part.
This structured representation allows for a comprehensive understanding of the task, bridging the gap between high-level instructions and low-level robotic execution. It moves beyond simply knowing which parts go together to understanding exactly *how* they connect.
Leveraging Vision-Language Models for Information Extraction
The core of Manual2Skill++’s ability to understand manuals lies in its use of advanced Vision-Language Models (VLMs), specifically Gemini-2.5-Pro. These models are trained to interpret both visual information (like diagrams and annotations in manuals) and text. The process involves a two-stage prompting strategy:
- First, the VLM analyzes a manual image to estimate the number and type of connectors needed for a particular step and which components they attach to.
- Second, using this high-level information along with high-fidelity 2D images of the components (rendered from 3D models with labeled attachment points), the VLM identifies the exact pairs of attachment points that need to be linked by each connector.
This iterative process, applied across all assembly steps, builds the complete connection-enriched hierarchical graph. This method significantly reduces the complexity of matching attachment points, which would otherwise be a daunting combinatorial problem given the visual clutter and minimal detail often found in manual sketches.
Achieving Millimeter-Level Precision in Part Alignment
One of the critical challenges in robotic assembly is accurately estimating the poses (position and orientation) of parts. Traditional learning-based methods often result in centimeter-level errors, which are too large for precise operations like inserting a dowel into a hole. Manual2Skill++ tackles this by using a direct connection-matching approach.
Since the system knows the precise attachment points and their correspondences from the extracted manual information, it can calculate the relative poses between connected components through geometric constraint satisfaction. This involves an optimization problem that ensures attachment positions coincide and their normal vectors are collinear and opposite. This constraint-based method achieves millimeter-level alignment accuracy, which is crucial for reliable connection operations.
Robust Robotic Assembly Execution
To validate its effectiveness, Manual2Skill++ was tested in a simulation environment called Isaac Lab, featuring four complex, long-horizon assembly tasks: an IKEA chair, a shoe shelf, an airplane model, and a LEGO figure. These tasks involve diverse connection mechanics, including mortise-tenon joints, dowels, and screws, with dedicated simulation mechanics for each type.
The framework successfully guided robots through these tasks, demonstrating that its hierarchical graph representation correctly determines the assembly sequence and that its connection-constrained pose alignment provides the necessary precision for robust execution. Even with simple local search strategies, the system achieved high success rates, especially when using a force-position hybrid strategy that detects contact forces to guide fine insertion.
The researchers also physically replicated all four benchmark tasks using real IKEA products and 3D-printed models, confirming that the simulation faithfully mirrors real-world assembly procedures. This ensures that the benchmark serves as a reliable testbed for developing and evaluating new methods for connector-aware manipulation.
Also Read:
- FALCON: Improving Robot Dexterity Through Advanced Spatial Perception
- DIV-Nav: Guiding Robots with Spatial Understanding for Multi-Object Search
The Future of Robotic Assembly
Manual2Skill++ represents a significant step forward in robotic assembly, reintroducing the importance of understanding and executing physical connections. By providing a framework that extracts structured connection information from instruction manuals and uses it for precise part alignment, it lays the groundwork for more generalizable and robust assembly systems. This work offers both the motivation and the tools for the research community to tackle more authentic, contact-rich assembly challenges. You can read the full research paper here.
