TLDR: This research paper reviews and classifies how large language models (LLMs) and vision-language models (VLMs) are integrated into robotic systems to enable more autonomous and interactive behaviors. It proposes a taxonomy based on four integration approaches (protocol, interface, orchestration, embedded) and six agent roles (planner, orchestration, task-specific, model-centric, generalist, generalist systemic), highlighting the shift towards AI agents that interpret human intent, plan tasks, and manage robot capabilities. The paper covers both academic and community-driven projects, emphasizing the rapid evolution and future potential of embodied agentic AI in robotics.
The field of robotics is undergoing a significant transformation with the rise of advanced artificial intelligence models, particularly large language models (LLMs) and vision-language models (VLMs). These models are enabling robots to interact with humans and their environments in increasingly sophisticated ways, moving beyond rigid programming to more flexible and intelligent behaviors.
Traditionally, robots were programmed with specific instructions for every task. However, the emergence of foundation models has paved the way for “agentic” AI in robotics. This means that instead of direct, step-by-step commands, robots can now interpret natural language instructions, plan their own actions, and even learn new skills. This approach allows robots to be more adaptable and user-friendly, as they can understand human intent and manage their own software tools without needing to discard existing, well-tested robot functionalities.
Understanding Agentic AI in Robotics
The concept of an embodied robotic agent, where intelligent behavior comes from a robot’s physical presence and interaction with its surroundings, has been around for decades. However, the recent advancements in LLMs, especially since the release of ChatGPT, have dramatically expanded what these agents can do. Early efforts focused on simple interfaces, like using LLMs to augment command-line tools for robot operating systems (ROS).
More recently, systems like ROSA, RAI, and BUMBLE have emerged, acting as intelligent intermediaries between human commands and robot actions. These frameworks allow robots to process open-ended instructions, integrate visual information, and execute complex tasks. For instance, ROSA translates natural language into validated robot actions, while RAI focuses on a flexible multi-agent framework where different AI agents collaborate for tasks like perception, planning, and motion control. BUMBLE, on the other hand, excels in mobile manipulation across large environments by combining perception, memory, and motor skills.
A newer development involves Model Context Protocol (MCP) servers, which act as plugins, allowing AI assistants like Claude to interact with existing robot applications. This approach leverages the built-in capabilities of these AI assistants, such as web search and code execution, to enable more advanced robot behaviors without extensive additional programming.
How Foundation Models Integrate with Robots
The research paper categorizes the integration of these powerful AI models into robotic systems into four main approaches:
- Protocol Integration: Here, the AI model acts primarily as a translator. It converts user input, often in natural language, into specific commands or tool calls that the robot understands. An example is a system that turns a spoken command into a ROS 2 command-line instruction.
- Interface Integration: This goes a step further by making the interaction more dynamic and interactive. The AI model not only translates commands but also uses feedback from the robot’s actions in the real world to inform future decisions. These systems often run in a continuous loop, allowing for more complex and iterative task execution. OpenMind OM1 is a notable example, aiming to be a decentralized AI-centric “operating system” for robots.
- Orchestration-Oriented Integration: In this approach, the AI model takes on a management role, overseeing and coordinating various robot resources, tools, or even other AI agents. It acts as a high-level planner, delegating tasks to different subsystems. AutoRT, for example, uses an LLM to orchestrate a fleet of mobile manipulators.
- Direct or Embedded Integration: This is where the AI model directly produces actions for the robot, often in an end-to-end manner, or serves as a specific perception module. These are sometimes called “robotic foundation models” and are trained to map sensory inputs directly to robot actions.
Also Read:
- Understanding AI Assistants: A Deep Dive into OS Agents for Digital Device Control
- Navigating the Landscape of LLM-Based Data Science Agents: A Comprehensive Survey
Roles of Robotic Agents Powered by LLMs
Beyond integration, the paper also classifies robotic agents based on their functional roles:
- Planner Agents: These agents use LLMs to generate a sequence of high-level actions or skills for the robot. The LLM focuses on reasoning and breaking down tasks, while lower-level controllers handle the actual execution. Google’s SayCan is a prime example, where an LLM suggests actions, and a separate system evaluates their feasibility.
- Orchestration Agents: Similar to orchestration integration, these agents manage interactions between multiple skills, components, or even other robots. They perform decision-level control, deciding which skill to activate or which robot should perform a task.
- Task-Specific Agents: While designed for specific problems, these agents leverage LLMs to enhance their performance, allowing for zero-shot reasoning or dynamic planning without extensive prior training for every specific task.
- Model-Centric Agents: These agents use a single, unified model to process various inputs (images, language, robot’s own state) and directly produce actions. They aim for a more integrated approach to robot control.
- Generalist Agents: These represent a shift towards AI architectures that can handle multiple tasks and domains. A central reasoning model (often an LLM) flexibly interacts with various executable components, allowing the robot to generalize across different tasks. Voyager, which generates its own tools and builds a skill library, is a good illustration.
- Generalist Systemic Agents: These focus on creating reusable, modular frameworks for developing and managing LLM-based robotic systems. They emphasize a clean separation and easy composition of perception, reasoning, and action modules, simplifying the overall development process.
The past few years have seen an explosion of innovation in how AI agents are integrated into robotics. From sophisticated tools built on frameworks like LangChain to community-driven projects and industrial solutions, the clear trend is towards equipping robots with a higher-level “AI agent” layer. This layer allows them to understand human intentions and effectively manage their own software toolkit, making intelligent interaction with complex robotic systems more accessible and adaptable. For a deeper dive into the technical details and a comprehensive list of projects, you can refer to the full research paper here.
