TLDR: The paper introduces a novel 3D weakly supervised semantic segmentation method that significantly improves performance by generating high-quality pseudo-labels. It uses a Class-Aware Label Refinement (CALR) module to balance categories and a Geometry-Aware Label Refinement (GALR) module to integrate 3D geometric information. Additionally, a Self-Training with Label Propagation (STLP) strategy iteratively refines and expands labels to unlabeled regions, achieving state-of-the-art results on ScanNet and S3DIS datasets with minimal annotation effort.
3D semantic segmentation, a crucial technique for understanding three-dimensional scene data, has seen remarkable advancements with deep learning. However, these powerful methods traditionally demand extensive, point-by-point annotations, a labor-intensive and costly process. This challenge has spurred the development of 3D weakly supervised semantic segmentation (3D WSSS), which aims to achieve accurate segmentation using only sparse or low-cost annotated data, such as scene-level labels.
Previous approaches to 3D WSSS often relied on class activation maps or pre-trained vision-language models. While promising, these methods faced significant hurdles: the pseudo-labels generated were often of low quality, and the inherent 3D geometric information of the scenes was not fully utilized. These limitations created bottlenecks, preventing the development of truly high-performance 3D WSSS models.
A new research paper, titled 3D Weakly Supervised Semantic Segmentation via Class-Aware and Geometry-Guided Pseudo-Label Refinement, introduces an innovative and effective method to overcome these challenges. Authored by Xiaoxu Xu, Xuexun Liu, Jinlong Li, Yitian Yuan, Qiudan Zhang, Lin Ma, Nicu Sebe, and Xu Wang, the proposed methodology integrates 3D geometric priors with a class-aware guidance mechanism to generate highly accurate pseudo-labels.
The Core Methodology
The new approach operates through two main stages: Pseudo Label Generation and Refinement, and Self-Training with Label Propagation. The first stage focuses on creating high-fidelity point-level pseudo-labels using only scene-level supervision, which is significantly less demanding than full annotation.
The Pseudo Label Generation and Refinement procedure involves two key modules:
- Class-Aware Label Refinement (CALR): This module addresses the problem of class imbalance. Instead of simply selecting the most confident pseudo-labels globally (which often favors larger categories like floors and walls), CALR selects a balanced proportion of high-confidence labels for each category. This ensures that smaller or rarer objects are not overlooked, leading to more balanced and accurate supervision for model training.
- Geometry-Aware Label Refinement (GALR): Recognizing the underutilization of 3D geometric information in prior methods, GALR integrates implicit 3D geometric constraints. It uses a technique called ‘superpoint analysis,’ where nearby points with similar geometric features are grouped. If a dominant category is identified within a superpoint with high confidence, that category is assigned; otherwise, the points remain unlabeled, effectively filtering out low-confidence and geometrically implausible pseudo-labels.
Even after these refinements, significant regions may still remain unlabeled. This is where the second stage, Self-Training with Label Propagation (STLP), comes into play. This iterative process continuously enhances pseudo-label quality and expands label coverage. The STLP module combines a Label Update strategy with the GALR module. The Label Update strategy gradually propagates pseudo-labels to previously unlabeled regions, retaining historical labels while incorporating new, reliable predictions. This iterative self-training process allows the model to progressively learn from its own refined predictions, fostering the development of robust 3D WSSS models.
Performance and Generalization
The effectiveness of this methodology has been rigorously validated through comprehensive experiments on two widely-used benchmarks: ScanNet and S3DIS datasets. The results demonstrate that the proposed method achieves state-of-the-art performance, significantly outperforming previous approaches that rely on similar or even richer supervision signals.
Notably, the method also exhibits remarkable generalization capabilities. When extended to unsupervised settings—meaning it operates without scene-level labels for filtering during pseudo-label generation and self-training—it still maintains competitive accuracy. This highlights the robustness of the CALR and GALR strategies in refining pseudo-labels and the scalability of the Label Propagation mechanism.
Also Read:
- ∇-SDF: Building Precise 3D Maps for Robots with Hybrid AI
- Unlocking 3D Spatial Reasoning in AI: Introducing 3DThinker
Future Directions
While the method marks a significant leap forward, the authors acknowledge certain limitations. The reliance on scene-level labels as filter masks still presents challenges in fully guiding the model to perceive scene categories. Furthermore, the integration of 3D geometric priors is currently indirect, leveraging superpoint information. Future research will explore ways to directly incorporate 3D geometric knowledge into the model, promising even greater advancements in 3D semantic segmentation.
