TLDR: BiBo is a novel framework that enables off-the-shelf Vision-Language Models (VLMs) like GPT-4 to control humanoid robots. It bypasses the need for extensive data collection by using an embodied instruction compiler to translate high-level commands into low-level actions, and a diffusion-based motion executor to generate smooth, environmentally adaptive movements. BiBo achieves high success rates in complex interaction tasks and significantly improves motion execution precision.
Humanoid robots are designed to interact with the physical world, but they often face significant challenges in handling diverse and flexible interactions within open environments. Traditionally, this problem has been addressed by collecting vast datasets to train highly capable models, a process that is both expensive and time-consuming. However, a new framework called BiBo (Building humanoId agentByOff-the-shelf VLMs) offers an innovative alternative by empowering existing Vision-Language Models (VLMs), such as GPT-4, to control humanoid agents.
The core idea behind BiBo is to leverage the strong open-world generalization capabilities of off-the-shelf VLMs, thereby reducing the need for extensive data collection. This approach is conceptually similar to how a computer operates, with a compiler and an assembler working together to execute tasks specified by programming languages.
How BiBo Works: Two Key Components
BiBo consists of two main components that bridge the gap between high-level human instructions and the low-level motor control required for physical humanoid interactions:
1. Embodied Instruction Compiler: This component acts like a translator. It enables the VLM to perceive its environment and accurately convert high-level user instructions (e.g., “have a rest”) into precise, low-level primitive commands with specific control parameters (e.g., “sit casually, location: (1, 2), facing: 90°”). The compiler achieves this through a three-stage visual question-answering (VQA) process. First, it analyzes basic attributes of the desired motion, such as the motion caption and target object. Next, it reasons about the agent’s pose, including its location and facing direction, by transforming complex numerical coordinates into a visual identification task that VLMs are better equipped to handle. Finally, it generates the target positions for key joints, often by placing a grid of labels over an image of the target object and allowing the VLM to select an anchor point.
2. Diffusion-based Motion Executor: Once the compiler generates a structured command, the motion executor takes over. This component is responsible for generating human-like motions from these commands. It uses a Latent Diffusion Model (LDM) and a Variational Autoencoder (VAE) to create smooth, continuous, and physically plausible movements. A crucial aspect of the executor is its ability to dynamically adapt to physical feedback from the environment. If, for instance, a collision occurs, the executor adjusts future motions based on the actual executed movement, ensuring continuity and environmental awareness. This prevents jerky movements and allows the humanoid to interact naturally with its surroundings.
Also Read:
- Orchestrating Robot Skills: MAESTRO’s Modular Path to Generalist AI
- URDF-Anything: Automating Digital Twin Creation for Articulated Objects
Impressive Results and Capabilities
Experiments have shown that BiBo is highly effective. It achieved an interaction task success rate of 90.2% in diverse, open environments, even when using an off-the-shelf VLM like GPT-4o. Furthermore, BiBo improved the precision of text-guided motion execution by 16.3% compared to previous methods. The framework can handle not only basic interactions but also complex and diverse motions, enabling infinite long-sequence synthesis and real-time interactive control through user instructions.
BiBo represents a significant step forward in embodied AI, demonstrating the potential of general-purpose VLMs to control humanoid agents without the prohibitive costs and difficulties associated with extensive data collection and task-specific training. This work opens new avenues for developing more versatile and adaptable humanoid robots in the physical world. For more detailed information, you can read the full research paper here.
