TLDR: ComposableNav is a new robot navigation system that uses composable diffusion models to allow robots to follow complex, multi-part instructions in dynamic environments. It learns individual motion primitives (basic skills) through a two-stage training process (supervised pre-training and reinforcement learning fine-tuning) and then composes them at deployment time to satisfy novel combinations of specifications. This approach significantly reduces complexity and enables robots to handle diverse, unseen instructions effectively in both simulations and real-world scenarios.
Robots are increasingly becoming a part of our daily lives, and a key challenge for their widespread adoption is enabling them to navigate complex, dynamic environments while following human instructions. Imagine telling a robot to “overtake the pedestrian while staying on the right side of the road.” This single instruction contains multiple specifications, and as robots gain more capabilities, the number of possible instruction combinations grows exponentially, making it incredibly difficult to program or train them for every scenario.
A new research paper introduces a novel solution called ComposableNav, which tackles this challenge by leveraging the power of diffusion models. The core idea behind ComposableNav is that following an instruction involves independently satisfying its individual components, or “specifications,” each corresponding to a distinct basic motion skill, known as a motion primitive.
How ComposableNav Works
Instead of trying to train a single, massive model to handle every conceivable instruction combination, ComposableNav learns each motion primitive separately. For example, it might learn primitives like “pass a person from the left,” “yield to a person,” or “walk through a specific region.” The magic happens at deployment time: when given a complex instruction, ComposableNav composes these learned primitives in parallel to generate a trajectory that satisfies all the specifications simultaneously, even if it’s a combination it has never encountered during training.
This approach dramatically simplifies the problem, reducing the complexity from exponential to linear. A relatively small set of motion primitives can support a vast, combinatorially large space of instructions, allowing users to customize robot behaviors in ways that align with human preferences and social interactions.
To avoid the laborious process of collecting demonstration data for each individual motion primitive, ComposableNav employs a clever two-stage training procedure:
- Supervised Pre-training: First, a base diffusion model is pre-trained using general-purpose navigation data. This data helps the robot learn to generate diverse, collision-free, and goal-reaching trajectories in dynamic environments.
- Reinforcement Learning Fine-tuning: In the second stage, this pre-trained base model is fine-tuned separately for each motion primitive using reinforcement learning (RL). For each primitive, a simple rule-based reward function is designed to evaluate how well a generated trajectory aligns with the instruction (e.g., did it successfully pass from the left?). This allows the robot to learn specific behaviors without needing explicit demonstrations for every single primitive.
At deployment, ComposableNav models the desired motion trajectory as a conditional distribution. It composes the relevant motion primitives by summing the predicted noise from each diffusion model’s denoising network during the trajectory generation process. This effectively guides the robot’s path to satisfy all specified instructions simultaneously.
Also Read:
- Scene Graphs Enable Robots to Master Complex Tasks with Focused Learning
- Smarter Navigation: How AI Agents Learn to ‘Walk and Read Less’ for Efficiency
Real-World Performance
The researchers evaluated ComposableNav through extensive simulations and real-world experiments using a Clearpath Jackal robot. In simulations, ComposableNav consistently outperformed existing VLM-based (Vision-Language Model) policies and costmap-composing baselines, especially as the complexity of instructions increased. While baselines struggled with multiple specifications, ComposableNav maintained high success rates, demonstrating its robustness in following complex, unseen instruction combinations.
In real-world tests, ComposableNav was deployed on a robot navigating scenarios like a narrow doorway and an open outdoor space. It achieved consistently high success rates, proving its effectiveness in practical settings. The system also demonstrated real-time performance, with initial trajectory generation taking around 0.4 seconds for the most complex cases and replanning requiring only 0.06 seconds, all on onboard hardware.
This work represents a significant step towards more adaptable and user-friendly robots that can seamlessly integrate into human environments by understanding and executing nuanced instructions. For more technical details, you can refer to the full research paper here.
