TLDR: GR-3 is a new large-scale vision-language-action (VLA) model from ByteDance Seed, designed for generalist robots. It excels in generalizing to novel objects, environments, and abstract instructions, and can be efficiently fine-tuned with minimal human data. Integrated with the versatile ByteMini robot, GR-3 demonstrates robust performance in long-horizon and dexterous tasks like pick-and-place, table bussing, and cloth manipulation, significantly outperforming previous state-of-the-art methods through its unique multi-faceted training approach.
In the ever-evolving world of artificial intelligence and robotics, the quest for intelligent generalist robots capable of assisting humans in daily life has been a long-standing ambition. A new technical report introduces GR-3, a significant stride towards achieving this goal. Developed by ByteDance Seed, GR-3 is a large-scale vision-language-action (VLA) model designed to empower robots with exceptional capabilities.
What is GR-3?
GR-3 stands out as a sophisticated VLA model that allows robots to understand and execute complex instructions, even those involving abstract concepts. It demonstrates remarkable ability to generalize to new objects, environments, and instructions it hasn’t encountered before. One of its key strengths is its efficiency: GR-3 can be fine-tuned with minimal human trajectory data, making it quick and cost-effective to adapt to new tasks and settings. Furthermore, it excels in handling long and intricate tasks, including those requiring two-handed manipulation and mobile movement, showcasing reliable and robust performance.
The model operates by taking natural language instructions, observations from its environment, and its current state as inputs. It then generates an action chunk to control a bi-manual mobile robot in an end-to-end manner. GR-3 is built upon a pre-trained vision-language model, specifically Qwen2.5-VL-3B-Instruct, and predicts actions using a technique called flow-matching.
A Comprehensive Training Approach
The impressive capabilities of GR-3 are a result of a multi-faceted training strategy. This includes co-training with vast amounts of web-scale vision-language data, efficient fine-tuning using human trajectory data collected via virtual reality (VR) devices, and effective imitation learning from robot trajectory data.
-
Imitation Learning with Robot Data: GR-3 learns from real robot trajectories collected through teleoperation. A unique data collection scheduler ensures diversity in the data, covering various actions, object combinations, and background settings. An innovative feature, ‘task status,’ is incorporated to improve the robot’s ability to follow instructions, even enabling it to identify and reject invalid commands.
-
Co-Training with Vision-Language Data: To enhance its generalization abilities, GR-3 is jointly trained on both robot trajectories and a large curated dataset of vision-language data. This co-training allows the model to understand novel semantics and generalize to unseen objects and complex concepts in a zero-shot manner, meaning it can perform tasks without prior specific training for them.
-
Few-Shot Generalization with Human Data: Recognizing the cost and time involved in collecting robot data, GR-3 leverages human trajectory data collected efficiently using VR devices. This allows for rapid and cost-effective adaptation to new scenarios with very little data, demonstrating the model’s ability to learn from cross-embodiment data and transfer those skills to the robot.
Introducing the ByteMini Robot
Alongside GR-3, the report introduces ByteMini, a versatile 22-degrees-of-freedom bi-manual mobile robot. Designed with flexible manipulation, high reliability, and user-friendliness in mind, ByteMini is the platform on which GR-3’s policies are deployed. Its unique sphere wrist joint configuration allows for human-like dexterity and operation in confined spaces. The robot also features an omni mobile platform for spatial mobility and height adjustment, and actuators based on the Quasi Direct Drive (QDD) principle for stability. Equipped with RGBD cameras on its head and wrists, ByteMini provides the necessary visual input for GR-3 to perform fine manipulation.
Real-World Performance
Extensive real-world experiments were conducted across three challenging tasks, where GR-3 consistently outperformed the state-of-the-art baseline method, π0:
-
Generalizable Pick-and-Place: GR-3 showed superior generalization to unseen objects, environments, and complex instructions. For instance, it significantly boosted the success rate in tasks involving unseen objects and instructions requiring abstract concept understanding (e.g., “put the largest object into the carton”). With just 10 human trajectories per object, its success rate on unseen objects improved from 57.8% to 86.7%.
-
Long-Horizon Table Bussing: In this multi-step task involving packing food, sorting utensils, and disposing of trash, GR-3 demonstrated robust performance and exceptional instruction following, even when faced with novel destinations or invalid commands. It achieved a 97.5% success rate in instruction following compared to π0’s 53.8%.
-
Dexterous Cloth Manipulation: GR-3 proved proficient in handling deformable objects, such as hanging clothes on a drying rack. It achieved high average task progress in basic settings, challenging positions, and with unseen clothes, showcasing its ability to manage complex dexterous skills.
Also Read:
- VIDAR: Advancing Bimanual Robot Control with Video Diffusion Models
- Gaze-Guided Robots: Enhancing Efficiency and Robustness with Human-Inspired Vision
Looking Ahead
While GR-3 represents a significant leap forward, the researchers acknowledge limitations, such as occasional struggles with novel concepts, unseen object shapes, and recovery from out-of-distribution states. Future work aims to scale up the model and training data, and incorporate reinforcement learning to further enhance robustness and performance.
In conclusion, GR-3 is a powerful vision-language-action model that brings us closer to the vision of generalist robots assisting humans in daily life. Its innovative training recipe and robust performance across diverse, challenging tasks mark a pivotal step in robotics. For more details, you can read the full research paper here.
