TLDR: Kitchen-R is a novel benchmark that unifies the evaluation of high-level task planning and low-level robot control in a simulated kitchen environment. It uses the Isaac Sim simulator, features a mobile manipulator robot, and includes over 500 complex language instructions. The benchmark supports independent assessment of planning and control, as well as crucial integrated evaluation of the entire system, bridging a key gap in embodied AI research.
Robotics and embodied AI are rapidly advancing fields, but a significant challenge has been the disconnect between how we evaluate high-level task planning and low-level robot control. Many benchmarks for language instruction following assume that a robot can perfectly execute basic actions, while those for low-level control often rely on very simple, one-step commands. This makes it difficult to assess how well an entire robotic system performs when both understanding complex instructions and physically executing them are crucial.
To address this critical gap, researchers have introduced Kitchen-R, a new benchmark designed to unify the evaluation of both task planning and low-level control. Imagine a robot in a simulated kitchen environment, a ‘digital twin’ of a real one, tasked with following complex instructions like ‘Move the red cup from the table to the shelf.’ Kitchen-R provides just such a scenario, built using the Isaac Sim simulator and featuring a mobile manipulator robot capable of moving around and interacting with objects.
The benchmark comes with over 500 intricate language instructions, allowing for a comprehensive test of a robot’s ability to understand and act upon human commands. What makes Kitchen-R particularly innovative is its flexible framework, offering three distinct evaluation modes:
Also Read:
- Enabling AI to Understand Evolving Human Preferences
- Unpacking LLM Reliability in Complex Scheduling: Introducing R-ConstraintBench
Three Ways to Evaluate Robot Intelligence
- Independent Planning Assessment: This mode focuses solely on how well a robot’s ‘brain’ (its planning module) can break down a complex language instruction into a series of executable steps.
- Independent Control Policy Assessment: Here, the focus shifts to the robot’s ‘body’ (its low-level control policy). Given a perfect plan, how well can the robot physically execute each step, navigating and manipulating objects in the simulated environment?
- Integrated System Evaluation: This is the most crucial mode, assessing the entire system end-to-end. It evaluates how well the planning module and the control policy work together, from understanding a complex instruction to successfully completing the physical task.
Kitchen-R also provides baseline methods to help researchers get started. For task planning, it uses a strategy based on a vision-language model (VLM), which can interpret both visual information and language. For low-level control, it employs a diffusion policy, a modern approach for generating smooth and effective robot movements. Additionally, the benchmark includes a system for collecting robot trajectories, which is vital for training and improving these policies.
The development of Kitchen-R is a significant step forward for embodied AI research. By offering a unified testbed and baselines, it enables more holistic and realistic benchmarking of language-guided robotic agents. This means researchers can now better understand how planning errors might interact with execution challenges, leading to the development of more robust and capable robots for real-world applications. For more in-depth technical details, you can refer to the full research paper here.
The benchmark has already proven its utility, having been successfully used for data collection and validation in the Embodied AI track of the AIJ Contest 2024, collecting approximately 2,700 mobile manipulation trajectories and over 500 diverse planning language instructions. This demonstrates its practical relevance and potential to drive future advancements in the field.
