TLDR: A new robotics research introduces “Compose by Focus,” a method that uses scene graphs to represent visual information for robots. By focusing only on task-relevant objects and their relationships, robots can learn and combine basic “atomic skills” more robustly, overcoming challenges in cluttered environments and achieving high success rates in complex, long-horizon manipulation tasks in both simulations and the real world.
Generalist robots that can perform a wide array of complex tasks are a long-standing goal in artificial intelligence and robotics. A key challenge in achieving this is enabling robots to combine basic, individual actions, known as atomic skills, to solve longer, more intricate tasks. While much research has focused on how to plan the sequence of these skills, ensuring that the individual skills themselves execute reliably in varied and cluttered environments has remained a significant hurdle.
A new research paper titled Compose by Focus: Scene Graph-based Atomic Skills by Han Qi, Changhe Chen, and Heng Yang introduces an innovative approach to address this problem. Their method, dubbed “Compose by Focus,” tackles the issue of visuomotor policies (robot control based on visual input) failing when faced with distribution shifts—meaning, when the real-world environment differs significantly from the simpler settings in which the skills were initially trained.
The Problem with Clutter
Imagine a robot trained to pick up a single carrot from a clean table. If that same robot is then asked to pick up all vegetables from a cluttered table with many other objects and distractors, its policy might fail. This isn’t necessarily because the high-level planner (which breaks down the task into “pick up carrot,” “pick up eggplant,” etc.) is flawed, but because the low-level visuomotor policy lacks the robustness to handle the visual complexity and irrelevant variations in the scene. Current state-of-the-art imitation learning methods often struggle with such compositional tasks due to brittle visual processing.
Scene Graphs: Focusing on What Matters
The core idea behind “Compose by Focus” is that for skills to be truly composable, they must be “focused.” This means the robot should only pay attention to the elements in the scene that are relevant to the current skill, ignoring everything else. To achieve this, the researchers propose using a scene graph-based representation of visual data.
Instead of processing raw RGB images or 3D point clouds, the robot’s visual input is transformed into dynamic semantic 3D scene graphs. These graphs are designed to be “relevant-object centric.” They include:
- Nodes: These represent task-relevant objects, such as the robot’s gripper, the object to be manipulated, the target location, and any necessary obstacles. These objects are identified and segmented from the visual input using advanced visual foundation models like Grounded-SAM, and their 3D geometry is encoded.
- Edges: These capture the dynamic relationships between objects, such as ‘grasp,’ ‘next to,’ or ‘inside.’ These relationships are inferred using powerful Vision-Language Models (VLMs) like ChatGPT.
By constructing these concise subgraphs, the system effectively filters out irrelevant visual noise, allowing the robot to concentrate on the crucial context for the task at hand.
Learning and Composing Skills
Once the scene graph is constructed, it’s fed into a Graph Neural Network (GNN) to extract meaningful features. These structured features, combined with a language description of the atomic skill (encoded by a CLIP encoder), are then used to train a diffusion-based visuomotor policy. This single policy learns all the individual atomic skills.
During testing, for a long-horizon task (like “pick up all vegetables and put them in the basket”), a VLM (e.g., ChatGPT-4V) acts as a high-level planner, breaking the task into sequential sub-goals. For each sub-goal, a dynamic sub-scene graph is created, focusing only on the objects relevant to that specific sub-goal. The trained policy then uses this focused scene graph and the sub-goal description to predict the necessary actions.
Impressive Results in Simulation and Real-World
The “Compose by Focus” method was rigorously tested in both simulated and real-world manipulation tasks. In simulations involving complex multi-skill tasks like sorting objects by color, stacking blocks, or using tools while avoiding obstacles, the scene graph-based approach achieved substantially higher success rates (often above 78%) compared to state-of-the-art baselines, which frequently failed or performed poorly (below 50%).
Real-world experiments, including vegetable picking in cluttered scenes and tool usage with obstacle avoidance, further demonstrated the method’s effectiveness. The scene graph-based policy achieved success rates of 0.97 for vegetable picking and 0.9 for tool usage, significantly outperforming other methods. This highlights the improved robustness and compositional generalization of the approach, even when faced with distractors or previously unseen obstacles.
Also Read:
- Cognitive-Inspired AI: A New Method for Attention Management in Transformers
- MICA: Intelligent AI Assistants for Modern Industrial Operations
Advantages and Future Directions
This research demonstrates that scene graph representations are crucial for enabling robust multi-skill composition. They offer flexibility in handling varying numbers of objects and explicitly encode inter-object relationships, which helps in disambiguating subtle action differences. This approach also reduces the exponential data demands typically associated with training robots for long-horizon tasks, as it allows composition directly from policies trained on individual atomic skills.
While the method introduces some computational overhead due to its reliance on VLMs for scene graph construction and can be affected by occasional inaccuracies from visual foundation models like Grounded-SAM, these are manageable limitations. Future advancements in VLMs and visual foundation models are expected to further enhance the speed and accuracy of this promising pipeline, paving the way for more capable and adaptable generalist robots.
