TLDR: A new method for 3D object segmentation uses “Granularity-Consistent automatic 2D Mask Tracking” to ensure consistent object boundaries across video frames, preventing conflicting labels. Combined with a three-stage learning process, it achieves state-of-the-art accuracy and can identify objects from diverse text descriptions, even for rare or complex items, without needing manual 3D annotations.
3D instance segmentation, a crucial task in computer vision and robotics, involves dividing 3D scenes into meaningful object segments. Traditionally, this has relied on extensive and costly manual 3D annotations, limiting its application to a narrow range of predefined object categories.
Recent advancements have explored generating pseudo-labels by transferring 2D masks from powerful foundation models to 3D. However, a significant challenge with these methods is their tendency to process video frames independently. This often leads to inconsistent segmentation granularity and conflicting 3D pseudo-labels, ultimately reducing the accuracy of the final segmentation.
Researchers Juan Wang, Yasutomo Kawanishi, Tomo Miyazaki, Zhijie Wang, and Shinichiro Omachi have introduced a novel approach to overcome these limitations. Their work, detailed in the paper Class-agnostic 3D Segmentation by Granularity-Consistent Automatic 2D Mask Tracking, proposes a “Granularity-Consistent automatic 2D Mask Tracking” method combined with a “three-stage curriculum learning framework.”
Addressing Inconsistent Segmentation
The core of their solution lies in maintaining temporal correspondences across video frames. Unlike previous methods that treat each frame in isolation, this new approach automatically tracks 2D masks, ensuring that the segmentation of an object remains consistent in its level of detail and boundaries as it moves or is viewed from different angles across frames. This eliminates the problem of conflicting 3D pseudo-labels that arise when the same object is segmented differently in successive frames.
The method leverages the capabilities of the Segment Anything Model (SAM) for initial mask generation on keyframes and SAM2 for propagating these masks across video sequences. A robust object state management system is also incorporated, allowing the system to handle objects that temporarily disappear (e.g., due to occlusion) and reappear later, maintaining their identity and consistent tracking.
A Progressive Learning Journey
-
Stage 1: Fragmented Warm-up Training Initially, the model is trained on 3D pseudo-labels derived from 2D masks generated on individual keyframes. While these initial labels might still be fragmented, this stage helps the model establish basic object-level feature representations.
-
Stage 2: Granularity-Consistent Segmentation Learning Building on the first stage, the model is then fine-tuned using the temporally consistent 3D pseudo-labels generated by the 2D mask tracking policy. This crucial stage resolves cross-frame granularity inconsistencies and enables the model to learn robust correspondences across different views and over time.
-
Stage 3: Full-Scene Fine-Tuning Finally, the model undergoes further fine-tuning on complete 3D point clouds of the entire scene. This stage refines segmentation boundaries and enforces global geometric coherence, moving from a partial-view understanding to a holistic scene comprehension.
Also Read:
- Bridging Modalities: UniSOT’s Unified Approach to Object Tracking
- URDF-Anything: Automating Digital Twin Creation for Articulated Objects
Achieving State-of-the-Art Performance
Experimental results demonstrate the effectiveness of this new method. It successfully generated consistent and accurate 3D segmentations, achieving state-of-the-art results on standard benchmarks like ScanNet200 and ScanNet++. Notably, it maintains real-time inference speeds, making it practical for real-world applications.
Beyond quantitative metrics, the approach also exhibits strong open-vocabulary capabilities. This means it can identify and localize objects based on arbitrary natural language queries, even for fine-grained distinctions or rare “long-tail” categories not explicitly present in training datasets. For instance, it can accurately distinguish between “bottled water” and “coca cola” or identify “green comforter” with precise boundaries. It also performs well with out-of-vocabulary queries involving color, material, spatial, and functional descriptors, showcasing its potential for flexible human-robot interaction and diverse 3D semantic understanding tasks.
By addressing the critical issue of inconsistent pseudo-labels and employing a structured learning pipeline, this research significantly advances class-agnostic 3D instance segmentation, paving the way for more robust and adaptable computer vision systems.
