TLDR: RoMaP is a new 3D editing framework that allows precise and drastic modifications to specific parts of 3D Gaussian Splatting scenes. It achieves this through a robust 3D mask generation module (3D-GALP) that ensures consistent part segmentation across views, and a regularized Score Distillation Sampling (SDS) loss that guides edits to target regions while preserving overall scene coherence. This significantly enhances controllability in 3D content creation.
A new research paper titled “Robust 3D-Masked Part-level Editing in 3D Gaussian Splatting with Regularized Score Distillation Sampling” introduces a groundbreaking method for editing 3D digital content with unprecedented precision. Authored by Hayeon Kim, Ji Ha Jang, and Se Young Chun from Seoul National University, this work addresses significant challenges in local 3D editing, particularly within the rapidly evolving field of 3D Gaussian Splatting.
Recent advancements in 3D neural representations and generative models have made it easier to create high-quality 3D content. However, making precise, localized changes to specific parts of a 3D object has remained a complex task. This is especially true for 3D Gaussian Splatting, a technique that represents 3D scenes using numerous small, colored “Gaussians” (like tiny, soft spheres). The difficulties arise from inconsistent 2D part segmentations when trying to define specific areas, and the inherent ambiguity of a common optimization technique called Score Distillation Sampling (SDS) loss.
The researchers propose a novel framework called RoMaP, which stands for Robust 3D-Masked Part-level Editing. RoMaP is designed to enable precise and even drastic modifications to specific parts of a 3D Gaussian scene. It tackles the existing limitations through two main innovations.
Robust 3D Mask Generation
The first core component is a robust 3D mask generation module, featuring something called 3D-Geometry Aware Label Prediction (3D-GALP). Imagine trying to draw a precise outline around an object’s eye from many different angles. It’s hard because the eye might look different, or be partially hidden, depending on the viewpoint. Traditional methods often rely on 2D segmentation, which can be inconsistent across multiple views, leading to inaccurate or incomplete masks in 3D.
3D-GALP addresses this by using a sophisticated mathematical tool called spherical harmonics (SH) coefficients. This allows the system to model how a part’s label (e.g., “eye” or “nose”) might appear differently from various viewpoints, and also accounts for “soft-label” properties, where a Gaussian might belong to different parts depending on the view. This results in highly accurate and consistent part segmentations across all viewpoints, which is crucial for reliable local edits.
Also Read:
- New AI Framework Reconstructs Dynamic Human-Object Interactions from Single-Camera Video
- Advancing 3D Scene Understanding with Feed-forward Reconstruction Models
Regularized Score Distillation Sampling (SDS) Loss
The second key innovation is a regularized SDS loss. While standard SDS loss is used to guide 3D content generation from text prompts, it often struggles to make significant or unconventional modifications to specific parts. RoMaP enhances this by combining the standard SDS loss with additional “regularizers.”
One important regularizer is an L1 anchor loss, introduced through a method called Scheduled Latent Mixing and Part (SLaMP) editing. SLaMP generates high-quality 2D images of the part being edited, ensuring that modifications are confined strictly to the target region while maintaining the overall coherence of the scene. This provides a clear direction for the SDS optimization. Additionally, a technique called “Gaussian prior removal” allows for changes that go beyond the existing context, offering greater flexibility. Robust 3D masking further prevents unintended edits from spreading to other areas.
The experimental results demonstrate that RoMaP achieves state-of-the-art performance in local 3D editing. It works effectively on both reconstructed (existing) and newly generated Gaussian scenes and objects. This framework makes it possible to perform more robust and flexible part-level 3D Gaussian editing, even allowing for drastic and unconventional modifications like turning a nose into an “emerald nose” or giving someone different colored eyes, all while maintaining the global consistency of the 3D model.
This research marks a significant step forward in making 3D content creation more controllable and customizable, opening up new possibilities for various industries, including mixed reality and robotics. For more technical details, you can read the full paper available at arXiv:2507.11061.
