TLDR: A new framework called GUIDER is proposed that integrates Vision-Language Models (VLMs) and Language Models (LLMs) into a robot’s intent recognition system. This allows robots to better understand human goals from natural language prompts, filter relevant objects and locations, and autonomously assist users by navigating to desired areas and manipulating objects, significantly improving human-robot collaboration.
Human-robot collaboration is a rapidly evolving field, but for robots to truly be helpful, they need to quickly understand what a human wants to do, explain their reasoning, and assist in achieving those goals. This is a complex challenge, as robots often struggle with inferring human intent, leading to misunderstandings and increased effort for the human operator.
A recent research paper introduces an exciting advancement in this area by augmenting an existing framework called GUIDER. Previously, GUIDER was a probabilistic system that could predict human navigation and manipulation intents. It used various inputs like controller data, occupancy maps, and visual information to estimate what a human was trying to do. While GUIDER was good at real-time predictions, it still required the human to manually command the robot to complete tasks, meaning the loop between understanding intent and actually assisting wasn’t fully closed.
The new proposal aims to bridge this gap by integrating powerful artificial intelligence models: Vision-Language Models (VLMs) and text-only Language Models (LLMs). These models create what the researchers call a “semantic prior.” Think of this semantic prior as a smart filter that helps the robot focus on objects and locations that are relevant to a given mission, based on a natural language prompt from the operator.
Here’s how this enhanced GUIDER system works: First, the robot’s onboard camera captures an image. A vision module, using technologies like YOLO for object detection and the Segment Anything Model (SAM) for instance segmentation, identifies and segments objects, providing their class labels and image crops. At the start of a task, the human operator provides a mission prompt, which can be something as simple as “Please hand me the television remote.”
The pre-trained VLM then takes an image crop of a detected object and the mission prompt, and calculates how likely that object is to match the described goal. Additionally, a text-only LLM can rank the relevance of all detected object labels based on the mission context. These scores are then combined with GUIDER’s existing navigation and manipulation calculations. This fusion effectively suppresses objects and areas that are irrelevant to the mission, allowing the robot to concentrate on potential targets. If the combined confidence in an object or area exceeds a certain threshold, the robot can then autonomously navigate to the desired location and retrieve the object, or switch to a shared-autonomy mode where it assists the operator.
A key advantage of this system is its adaptability. The operator can modify the mission prompt at any time, and the robot’s understanding and subsequent actions will immediately adjust. This dynamic interaction is crucial for fluent human-robot collaboration.
The researchers plan to evaluate this VLM-based GUIDER in a simulated domestic living room environment using Isaac Sim. The robot, a Franka Emika Panda arm mounted on a Clearpath Ridgeback mobile base, will be tested on various tasks involving specific, categorical, and relational prompts (e.g., “Bring me the red mug,” “Pick up a drink,” or “Fetch the cup next to the laptop.”). The evaluation will measure how quickly the robot confidently predicts intent, the accuracy of its predictions, and the time it takes to complete assistance.
Also Read:
- Empowering Robots to Predict and Prevent Task Failures
- Navigating Complexity: How AI Language Models Are Enhancing Classical Planning
This work represents a significant step towards more intuitive and effective human-robot interaction. By leveraging the power of vision-language models, robots can move beyond simple commands to truly understand and assist humans in complex, real-world scenarios. The full research paper can be found here: Utilizing Vision-Language Models as Action Models for Intent Recognition and Assistance.
