TLDR: GeoSAM2 is a new framework that enables highly precise 3D part segmentation of textureless objects by casting the task as multi-view 2D mask prediction. It uses a modified SAM2 backbone with geometry-aware enhancements and a redesigned memory mechanism, allowing users to control segmentation with simple 2D clicks or boxes. The method achieves state-of-the-art performance, offers fine-grained control, and has broad applications in 3D editing and generation, all with minimal computational overhead.
A new research paper introduces GeoSAM2, an innovative framework designed to revolutionize 3D part segmentation. This method allows for incredibly precise and detailed segmentation of 3D shapes, controlled simply through 2D prompts like clicks or drawing boxes. It demonstrates strong generalization capabilities on a wide range of real-world objects, significantly outperforming existing methods while requiring minimal computational resources.
At its core, GeoSAM2 tackles the challenging task of 3D part segmentation by re-imagining it as a multi-view 2D mask prediction problem. For objects without texture, the system renders normal and point maps from various viewpoints. Users then provide simple 2D prompts on these rendered views to guide the selection of specific parts. These prompts are processed by a modified SAM2 backbone, a powerful 2D segmentation model, which is enhanced with a technique called LoRA (Low-Rank Adaptation) and a residual geometry fusion mechanism. This allows the system to reason about specific views while still leveraging the strong pre-trained knowledge of SAM2. The predicted 2D masks are then projected back onto the 3D object and combined across different views to create a coherent 3D segmentation.
One of the key advantages of GeoSAM2 is its ability to provide fine-grained, part-specific control without the need for text prompts, time-consuming per-shape optimization, or extensive 3D labels. Unlike methods that rely on global clustering or scale-based adjustments, GeoSAM2’s prompts are explicit, spatially grounded, and easy to interpret. This leads to more intuitive and accurate results.
Existing 3D part segmentation methods often face limitations. Some are fast but lack flexibility, offering only coarse control over segmentation. Others are expressive but very slow, making them impractical for real-time use. Furthermore, many methods struggle to fully align 2D interactions with precise 3D outcomes. GeoSAM2 addresses these gaps by reformulating the problem, enabling a new level of controllability and precision in understanding object parts in 3D.
The system’s design incorporates several clever techniques. It uses geometry-aware encoders with LoRA to efficiently adapt SAM2, originally trained on RGB images, to handle textureless 3D inputs like normal and point maps. A residual fusion strategy is employed to combine features from normal and point maps, ensuring that the model gradually learns cross-modal associations while preserving SAM2’s pre-trained knowledge. Crucially, GeoSAM2 redesigns SAM2’s memory mechanism. Instead of discarding older views, it retains features from all views throughout the sequence, ensuring cross-view consistency and preventing information loss, especially in multi-view 3D segmentation where viewpoint shifts are significant. A simple “frame repetition” bootstrapping strategy further enhances initial segmentation quality by providing an immediate memory prior.
Experimental results demonstrate that GeoSAM2 achieves state-of-the-art class-agnostic segmentation performance on challenging benchmarks like PartObjaverse-Tiny and PartNetE. It significantly outperforms both slow optimization-based pipelines and fast but coarse feedforward approaches. For instance, on PartObjaverse-Tiny, GeoSAM2 achieved an average mIoU of 84.06%, notably higher than PartField’s 79.18% and SAMesh’s 56.86%. On PartNetE, it scored 74.42%, maintaining a significant lead over other methods even when deprived of mesh connectivity priors. The method also shows strong generalization capabilities when applied to 3D models generated by generative models, maintaining clear part awareness even with blurred geometry boundaries.
The versatility of GeoSAM2 opens up broad applications. It can be seamlessly integrated with 3D part completion models, enabling effective 3D part amodal segmentation where users can dynamically adjust part granularity to generate clean, completion-ready segments. This capability is invaluable for creating artist-grade modular 3D structures. Furthermore, it allows for high-precision 3D part editing with minimal 2D inputs, enabling users to extract or merge components with pixel-level precision that propagates to 3D with well-defined boundaries.
Also Read:
- Enhancing Visual Perception with Decoupled Learning: Introducing DeCLIP
- Crafting Images with Layered Detail: Introducing Next Visual Granularity Generation
While GeoSAM2 represents a significant leap forward, the authors acknowledge a limitation: it may struggle with heavily occluded objects due to its multi-view nature. Future work could explore incorporating 3D-aware semantic completion models to address this. Overall, GeoSAM2 paves the way for efficient, accurate, and user-driven 3D segmentation in real-world applications, bridging the gap between 2D segmentation priors and 3D geometric reasoning. You can find the full research paper here: GeoSAM2: Unleashing the Power of SAM2 for 3D Part Segmentation.
