TLDR: LagMemo is a new robot navigation system that uses a language-enhanced 3D Gaussian Splatting memory to enable robots to navigate to multiple goals specified through various modalities (text, image, object category) in open-vocabulary environments. It builds a detailed 3D map with language features during exploration and then uses this memory for efficient goal localization and verified navigation, outperforming existing methods on a new benchmark called GOAT-Core.
Intelligent robots are increasingly expected to perform complex tasks in our homes and workplaces, requiring them to understand instructions, perceive their surroundings, and navigate to specific targets. While traditional visual navigation methods often struggle with diverse and unpredictable real-world scenarios, a new system called LagMemo is set to change this by enabling robots to navigate using a sophisticated language-enhanced 3D memory.
Most existing robot navigation systems are limited to finding a single goal, using a single type of input (like an object category), and operating within a predefined set of targets. However, real-world applications demand much more: robots need to understand goals described in various ways (text, images, or object categories), find multiple targets within the same environment, and identify objects not explicitly programmed beforehand. This is known as multi-modal, open-vocabulary, multi-goal visual navigation.
LagMemo, short for Language 3D Gaussian Splatting Memory, addresses these challenges by building a unified 3D language memory of its environment during an initial exploration phase. This memory isn’t just about geometry; it also stores rich language-based information about objects and areas. When given a new task, LagMemo queries this memory to predict potential goal locations and then uses a local perception system to verify and confirm targets as it navigates.
How LagMemo Works: Building a Smart Memory
The system operates in two main phases: memory reconstruction and memory-guided navigation.
Memory Reconstruction: During its initial exploration, the robot actively scans the environment, collecting visual data (RGB images, depth information) and its own position. This data is used to create a 3D Gaussian Splatting (3DGS) representation of the scene. Think of 3DGS as a collection of tiny, colored 3D points (Gaussians) that can accurately render the environment. To ensure the memory is robust even with sparse observations, LagMemo incorporates a keyframe retrieval mechanism, revisiting important past views to maintain reconstruction quality.
Crucially, LagMemo injects language features into this 3D geometric map. It uses advanced vision-language models like SAM (Segment Anything Model) to identify individual objects and CLIP (Contrastive Language-Image Pre-training) to extract high-dimensional language descriptions for these objects. These language features are then associated with the 3D Gaussians and organized into a ‘codebook’. This codebook allows LagMemo to understand and retrieve objects based on multi-modal queries, effectively creating a language-conditioned 3D memory.
Memory-Guided Navigation: Once the memory is built, LagMemo can tackle navigation tasks. When a goal is provided (e.g., “find the red mug” or an image of a specific chair), the system converts this query into a language embedding. It then compares this embedding to the entries in its codebook to identify candidate instances in the 3D memory. The geometric center of these candidate Gaussians is projected onto a 2D map, becoming a ‘waypoint’ for the robot.
The robot then plans a collision-free path to this waypoint. Upon reaching it, a critical ‘goal verification’ mechanism kicks in. Using models like SEEM for open-vocabulary segmentation and LightGlue for image matching, LagMemo confirms whether the observed object truly matches the goal. If confirmed, the robot proceeds to the exact stopping point; if not, it queries the memory for the next best candidate waypoint. This iterative process ensures reliable navigation even in complex, ambiguous situations.
Evaluating Performance with GOAT-Core
To rigorously test LagMemo, the researchers curated a new benchmark called GOAT-Core, a high-quality subset of the existing GOAT-Bench dataset. GOAT-Core features longer episodes with more subtasks, greater goal diversity, and increased distances between goals, pushing the boundaries of memory and long-term planning. It also addresses quality issues in the original dataset, ensuring fair and accurate evaluation.
Impressive Results
Experiments on GOAT-Core demonstrated LagMemo’s superior performance. In goal localization tasks, LagMemo achieved an overall 70.8% success rate, significantly outperforming baseline methods like VLMaps (58.8%) across object, image, and text modalities. This highlights the advantage of LagMemo’s detailed 3D spatial context.
For multi-modal multi-goal visual navigation, LagMemo consistently achieved the highest success rates across all tested environments, outperforming state-of-the-art baselines by a clear margin. It showed particular strength in text-based navigation tasks. Ablation studies confirmed that both the keyframe retrieval mechanism for geometry and the discrete language codebook for semantic association are crucial for the system’s effectiveness, as is the sophisticated goal verification module.
Also Read:
- HyPerNav: A New Approach for Robots to Find Objects Using Hybrid Perception
- Scaling Open-Vocabulary 3D Interaction: Introducing Gen-LangSplat for Efficient Language-Guided Scene Understanding
The Future of Robot Navigation
LagMemo represents a significant step forward in embodied AI, offering robots a powerful way to understand and navigate complex, dynamic environments using a rich, language-enhanced 3D memory. While the results are promising, future work aims to improve memory-aware exploration, enable incremental learning for dynamic environments, and optimize memory usage through pruning and compression techniques. For more details, you can read the full research paper here.
