TLDR: A new framework uses Large Language Models (LLMs) combined with spatial concept models to enable multiple robots to efficiently understand natural language commands, break them into subtasks, and assign them based on each robot’s learned knowledge of object locations. This approach significantly improves task allocation accuracy in both simulated and real-world multi-object retrieval scenarios compared to random or purely commonsense-based methods.
In the evolving world of robotics, enabling machines to understand and execute complex human commands, especially in multi-robot scenarios, remains a significant challenge. Imagine telling a team of robots, “I want to make a fruit smoothie, so please find an apple and a banana,” or “Get ready for a field trip.” These instructions require not only understanding ambiguous language but also efficiently coordinating multiple robots, each with its own limited knowledge of the environment.
A recent research paper, titled “Multi-Robot Task Planning for Multi-Object Retrieval Tasks with Distributed On-Site Knowledge via Large Language Models,” addresses this very problem. Authored by Kento Murata, Shoichi Hasegawa, Tomochika Ishikawa, Yoshinobu Hagiwara, Akira Taniguchi, Lotfi El Hafi, and Tadahiro Taniguchi, this study introduces a novel framework that significantly enhances how robot teams can collaborate to fulfill such commands.
The core issue is that robots often have “distributed, incomplete situational knowledge.” This means each robot might only know about its specific area—for instance, one robot might be familiar with the kitchen, while another knows the living room. The goal is to assign subtasks (like finding an apple) to the robot most likely to succeed, leveraging its local knowledge.
A Smart Approach to Robot Coordination
The proposed framework integrates Large Language Models (LLMs), such as GPT-4, with what the researchers call “spatial concept models.” These spatial concept models are crucial; they allow robots to learn and represent “where what is likely to be” in their environment. This includes understanding room names (like ‘kitchen’ or ‘bedroom’) and the probability of finding certain objects in those rooms, all derived from their observations.
The framework operates through a four-stage pipeline:
- Acquisition and Representation of On-site Knowledge: Each robot explores its designated area and builds its own spatial concept model. This model captures place-related vocabulary and object-to-location probability tables, essentially creating a detailed map of what it knows about its surroundings.
- Task Decomposition and Allocation: When a user gives a natural language instruction, the LLM first breaks it down into smaller, executable subtasks. Then, using the spatial knowledge from each robot, the LLM assigns these subtasks. For example, if Robot 1 has a high probability of finding apples in its area (the kitchen), it will be assigned the task of finding an apple.
- Sequential Action Planning: Once a robot receives its subtask, the LLM generates a step-by-step plan. This plan includes actions like navigation, object detection, picking up an object, and placing it. The planning prioritizes locations where the target object is most likely to be found, based on the robot’s spatial knowledge.
- Robot Execution and Feedback: The generated action plans are executed by the robots. If a step fails (e.g., an object isn’t detected or a grasp fails), the system provides immediate feedback to the LLM, which then attempts to replan or select alternative actions, ensuring robustness in dynamic environments.
This system can even handle ambiguous instructions. For instance, if asked to “Get ready for a field trip,” the LLM uses its common-sense reasoning, guided by specific examples, to infer necessary items like a “water bottle” or “backpack.” These inferred items are then converted into subtasks and allocated based on the robots’ spatial knowledge.
Putting it to the Test: Simulations and Real-World Scenarios
The researchers conducted two main experiments to validate their framework:
Experiment 1: Simulated Environment
A simulated two-floor home environment with 10 rooms and 24 objects was created. Two robots were deployed, each confined to a single floor, simulating a scenario where robots have distinct areas of responsibility. The goal was to see if the system could correctly assign subtasks based on each robot’s learned knowledge. The proposed method was compared against random allocation and a commonsense-only allocation (where GPT-4 relied solely on general knowledge without learned spatial concepts).
The results were striking: the proposed method achieved 47 out of 50 successful assignments, significantly outperforming random allocation (28/50) and GPT-4 Commonsense Allocation (26/50). This clearly demonstrated the value of integrating environment-specific spatial knowledge into the task assignment process.
Experiment 2: Real-World Demonstration
The framework was deployed on two Toyota Human Support Robots (HSRs) in a real-world household-like environment during the RoboCup @Home DSPL competition. The robots were given the ambiguous instruction: “Get ready for a field trip.”
The LLM successfully decomposed this command, assigning tasks like retrieving a “bag” and “snacks” to Robot 1, and a “water bottle” and “fruit juice” to Robot 2. Robot 2 successfully completed its assigned tasks, navigating, detecting, picking, and placing items. While Robot 1 encountered a physical issue (a grasped bag interfering with safety sensors), the experiment qualitatively confirmed the framework’s ability to interpret ambiguous instructions, decompose tasks, and allocate them effectively in a realistic, noisy environment.
Also Read:
- Advancing Robot Generalization Through Preserved Vision-Language Representations
- Robots Learn to Multitask: A New Approach for Handling Multiple Urgent Questions in 3D Environments
Looking Ahead
While highly effective, the framework has some limitations. It currently struggles with tasks requiring synchronous cooperation (like dual-arm manipulation or object handover) and lacks dynamic reallocation if a robot fails during execution. The study also used identical robots, whereas real-world scenarios often involve heterogeneous robot teams with varying capabilities. Future work aims to address these limitations by incorporating skill-aware allocation, dynamic reallocation, and extending the framework to human-robot collaboration.
This research marks a significant step towards more intelligent and cooperative multi-robot systems, enabling them to handle complex, real-world tasks with greater accuracy and robustness. For more details, you can read the full research paper here.
