TLDR: MoDeSuite is the first comprehensive task suite designed to benchmark robot learning for mobile manipulation of deformable objects. It features eight diverse tasks involving both elastic and plastic materials, supporting wheeled and legged robots, and various observation/action spaces. The suite enables systematic evaluation of reinforcement and imitation learning algorithms, demonstrating promising sim-to-real transfer for state-based policies while highlighting challenges in visual domain generalization.
Robots are becoming increasingly capable, but one area that remains a significant challenge is mobile manipulation, especially when dealing with objects that can change shape, known as deformable objects. Imagine a robot needing to pick up a soft towel, navigate a corridor while holding a flexible hose, or pull back a curtain. These tasks are complex because deformable objects behave unpredictably, unlike rigid items. While there are many benchmarks for robots handling rigid objects or manipulating deformable objects in a stationary setting, there hasn’t been a comprehensive, standardized way to test robots that combine both mobility and deformable object manipulation.
To bridge this crucial gap, researchers have introduced MoDeSuite, the first-of-its-kind Mobile Manipulation Deformable Object task suite. This new benchmark is specifically designed to push the boundaries of robot learning in scenarios involving flexible and pliable materials. MoDeSuite features eight distinct tasks, carefully crafted to mimic real-world challenges. These tasks involve both elastic objects, which temporarily change shape like rubber, and plastic deformable objects, which undergo more permanent changes, such as cloth or rope.
Success in MoDeSuite’s tasks demands sophisticated coordination between a robot’s mobile base (for movement) and its manipulator arm (for handling objects). It also requires the robot to intelligently use the object’s deformability to its advantage. The suite is built within Isaac Lab and leverages the high-fidelity Isaac Sim simulator, allowing for efficient training of robot learning algorithms in parallel environments.
MoDeSuite supports two main types of mobile manipulators: a wheeled robot configuration, which pairs a Franka Panda arm with a Ridgeback wheeled base, and a legged configuration, featuring the Boston Dynamics Spot robot with its arm. This diversity allows for testing algorithms across different robot morphologies. The suite also offers various ways for robots to perceive their environment, including image-based observations from RGB-D cameras and detailed state-based observations, and supports different action spaces for controlling the robot’s movements.
The Diverse Tasks of MoDeSuite
The eight tasks in MoDeSuite are inspired by everyday situations and are categorized into elastic and plastic deformation challenges:
- Elastic Tasks:
- Place: The robot must position an elastic rod onto a table beyond its arm’s reach, requiring coordinated movement of both the base and arm.
- Bend: The robot navigates an L-shaped corridor while holding a long elastic rod, bending it to fit through tight spaces.
- Transport: An extension of Bend, where the robot must navigate around a large obstacle while carrying the elastic rod.
- Drag: The robot manipulates an elastic belt fixed to a cube, lifting and stretching it over an obstacle while maintaining its body position.
- Lift: Inspired by tasks like operating a roller shutter, the robot lifts a suspended elastic belt to pass underneath it.
- Plastic Tasks:
- Uncover: The robot approaches a table and removes a table cover by pulling it, ensuring proper folding.
- Cover: The robot grasps a fabric and uses it to cover a gap between two objects.
- Curtain: The robot moves a hanging curtain aside and navigates its body through the opening without collisions.
Also Read:
- Charting the Course for AI in Robotics: Challenges and Future Directions
- Advancing Robot Dexterity in Fabric Handling with Semantic Keypoints
Benchmarking and Real-World Application
To demonstrate the utility of MoDeSuite, the researchers benchmarked four state-of-the-art learning algorithms: two reinforcement learning algorithms (Proximal Policy Optimization or PPO, and Soft Actor-Critic or SAC) and two imitation learning algorithms (Behavior Cloning and a retrieval-based method). The experiments showed that all tasks are solvable, yet challenging, for current algorithms, highlighting the need for further advancements.
A significant finding was the impact of robot morphology; tasks with the legged Spot robot were generally more challenging than those with the wheeled Franka setup, primarily due to the added complexity of maintaining balance. PPO consistently outperformed SAC across most tasks.
Crucially, MoDeSuite also facilitates the transfer of learned policies from simulation to the real world. Policies trained in simulation for tasks like Place and Drag (involving elastic objects) showed promising sim-to-real transferability when deployed directly on a physical Boston Dynamics Spot robot, especially when using state-based observations. However, for image-based policies, a noticeable gap was observed in tasks like Curtain, suggesting that visual differences between simulation and reality remain a challenge for direct transfer.
MoDeSuite represents a vital step forward in robot learning, providing a unified platform for research, benchmarking, and algorithm development in mobile manipulation with deformable objects. By bridging this gap, it aims to accelerate progress in both fields, paving the way for more adaptable and capable robots in diverse real-world environments. For more details, including code and videos, you can visit the project’s website: MoDeSuite Research Paper.
