TLDR: AnywhereVLA is a modular robotic system that enables mobile manipulators to understand natural language commands, explore unknown indoor environments, and perform pick-and-place tasks. It combines classical navigation (SLAM, semantic mapping, exploration) with a fine-tuned Vision-Language-Action (VLA) model for manipulation, running efficiently on onboard consumer hardware. The system achieved a 46% overall task success rate in unseen environments, demonstrating robust performance but with limitations in understanding precise spatial-semantic constraints in instructions.
Robotics is rapidly advancing beyond controlled factory settings, moving into complex, unpredictable indoor environments like homes and offices. Imagine a robot that can understand your natural language commands, explore an unfamiliar space, and then pick up and place objects with precision. This ambitious goal is being tackled by a new modular framework called AnywhereVLA.
Developed by a team of researchers including Konstantin Gubernatorov, Artem Voronov, Roman Voronov, Sergei Pasynkov, Stepan Perminov, Ziang Guo, and Dzmitry Tsetserukou, AnywhereVLA is designed for language-conditioned exploration and mobile manipulation in large-scale, previously unseen indoor environments. It aims to bridge the gap between robust, geometry-based navigation and the flexible, task-generalizing capabilities of language-conditioned manipulation.
What is AnywhereVLA?
At its core, AnywhereVLA is a sophisticated system that takes a simple text prompt from a user and translates it into a series of actions for a mobile robot. This isn’t just about moving from point A to point B; it’s about understanding complex instructions like “Pick up the blue box and place it in the designated area.”
The framework integrates several key technologies:
- Classical SLAM (Simultaneous Localization and Mapping): Using LiDAR and cameras, the robot builds a detailed 3D map of its surroundings while simultaneously figuring out its own position within that map.
- Metric Semantic Mapping: Beyond just geometry, the map also includes semantic information, meaning the robot understands what different objects are and where they are located.
- Task-Aware Frontier Exploration: If the target object isn’t immediately visible, the robot intelligently explores unknown areas, prioritizing regions where it’s most likely to find what it’s looking for.
- Approach Planner: Once an object is located, this module calculates the optimal position for the robot’s base to ensure the manipulator can reach and interact with the object effectively.
- SmolVLA Manipulation Head: For the actual grasping and placing, AnywhereVLA uses a compact Vision-Language-Action (VLA) model called SmolVLA. This model is fine-tuned on specific pick-and-place trajectories, allowing it to ground visual context and sub-goals into precise grasp and place proposals.
One of the impressive aspects of AnywhereVLA is its ability to run entirely onboard using consumer-level hardware. A Jetson Orin NX handles perception and VLA tasks, while an Intel NUC manages SLAM, exploration, and control, all while maintaining real-time operation.
The HermesBot Mobile Manipulator
To bring AnywhereVLA to life, the researchers developed a custom mobile manipulator platform called HermesBot. This robot features a two-wheeled differential drive base and is equipped with an SO-101 manipulator. Its sensor suite is comprehensive, including a Velodyne VLP-16 LiDAR and Intel RealSense D435i RGB-D cameras for navigation and SLAM. For the VLA subsystem, three Intel RealSense D435 cameras are strategically placed: a third-person-view camera, a wrist-mounted camera, and a base-mounted camera, providing a rich visual context for manipulation tasks.
Performance in Real-World Scenarios
AnywhereVLA was rigorously tested in diverse, unseen indoor environments, including a multi-room university lab with dynamic clutter and human presence. The robot was given commands like “Pick up the object and place it in the area. And bring the object to location.”
The system achieved an overall task success rate of 46%. Notably, the VLA Manipulation module, powered by the fine-tuned SmolVLA, boasted an 85% success rate, significantly outperforming systems without this specialized fine-tuning. In terms of speed, AnywhereVLA successfully completed tasks within a 5-meter exploration radius (comparable to an average apartment) in under 133 seconds on average, and within a 10-meter radius in under 10 minutes.
Also Read:
- Robots Learn to Navigate Like People: Introducing ReasonNav
- Meta-Memory: Empowering Robots with Advanced Spatial Reasoning Through Integrated Memory
Looking Ahead: Addressing Complex Instructions
While AnywhereVLA demonstrates robust performance in identifying and localizing target objects, the researchers acknowledge a current limitation: the system struggles with precise spatial-semantic constraints embedded in natural language instructions. For example, a command like “Pick up the bottle from the table and place it in the blue box” might lead the robot to pick up any bottle it finds, regardless of its initial location.
Future work aims to address this by incorporating hierarchical semantic parsing and relational reasoning modules. This could involve building a dynamic scene graph to understand relationships between objects (e.g., “bottle supported by table”) and augmenting exploration policies with graph-based reward signals. These enhancements will enable AnywhereVLA to follow human-like instructions more accurately in complex, multi-object environments.
For more technical details, you can read the full research paper here.
