TLDR: This research introduces a novel method to reconstruct complete 3D scenes, including occluded areas, from just a single input image using denoising diffusion models and Gaussian splatting. The key innovation is a Variational AutoReconstructor that learns a 3D latent space from 2D images without needing ground-truth 3D data. This generative approach allows for diverse and high-quality 360-degree renderings, outperforming existing methods in speed and completeness.
Imagine taking a single photograph and instantly generating a complete, interactive 3D model of the entire scene, including parts that were hidden from the camera’s view. This is the ambitious goal tackled by a new research paper titled Complete Gaussian Splats from a Single Image with Denoising Diffusion Models, authored by Ziwei Liao, Mohamed Sayed, Steven L. Waslander, Sara Vicente, Daniyar Turmukhambetov, and Michael Firman from the University of Toronto and Niantic Spatial.
Traditional 3D scene modeling techniques, like Gaussian splatting, are excellent at creating high-quality 3D representations. However, they typically demand a large number of images—often hundreds—to capture all the details and ensure a complete reconstruction. When observations are sparse, or only a single image is available, these methods struggle, leading to blurry, incomplete, or artifact-ridden models, especially in occluded areas or regions outside the camera’s immediate view.
The core challenge lies in the inherent ambiguity of reconstructing hidden surfaces. Conventional approaches often try to predict a single, ‘average’ version of the unseen parts, which can result in unrealistic or blurry outcomes. These methods usually focus on isolated objects or visible surfaces, failing to extrapolate effectively to fill in large unobserved areas.
The researchers propose a groundbreaking generative approach that uses a latent diffusion model to reconstruct a complete 3D scene with Gaussian splats from just one input image during inference. This model can even fill in the occluded parts, offering a significant leap forward in 3D scene understanding.
A Generative Leap Forward
Unlike regression-based methods that produce a single output, this new generative formulation learns a distribution of 3D representations. This means it can sample multiple plausible hypotheses for ambiguous or occluded regions, leading to much sharper and more realistic reconstructions. This ability to generate diverse samples is a key differentiator, allowing for a richer and more accurate representation of the real world.
The Variational AutoReconstructor: Learning from 2D Images
A major hurdle in training such a model is the lack of extensive ground-truth 3D training data for complete scenes. To overcome this, the team introduced a novel component called the Variational AutoReconstructor (VAR). This architecture learns a latent space for 3D Gaussian splats directly from 2D images in a self-supervised manner. Essentially, it encodes a 2D image into a compact latent code and then reconstructs a 3D representation, using differentiable rendering to compare the reconstructed 3D scene’s 2D projections with the original input images. This ingenious method bypasses the need for costly 3D ground truth data.
To ensure that fine-grained details are not lost during this compression, the VAR incorporates ‘skip connections,’ which allow high-frequency information to flow directly from the encoder to the reconstructor. This helps preserve the sharp textures and intricate details crucial for high-quality 3D models.
Denoising Diffusion Models for 3D
Once the Variational AutoReconstructor has learned this efficient latent space, a denoising diffusion model is trained on it. This model learns to generate these 3D latent codes, conditioned on a single input image. The process involves progressively adding noise to the latent codes and then training the model to reverse this process, effectively learning the underlying 3D distribution. The model also uses ‘classifier-free guidance’ during training, which provides flexible control over the balance between faithfulness to the input image and the diversity of the generated outputs.
Also Read:
- Navigating Complex Tasks with Tree-Guided Diffusion
- Boosting AI’s Digital Display Reading Accuracy with Synthetic Data
Real-time, High-Quality Results
During inference, the system takes a single RGB image, encodes it, and then uses the denoising diffusion model to generate a latent code. This code is then decoded by the AutoReconstructor into a Splatter Image, which is back-projected to create the final 3D Gaussian splat. This entire process is remarkably efficient, enabling real-time computation and high-quality 360-degree renderings.
Experiments on challenging datasets like CO3D and RealEstate10K demonstrate the method’s superior performance. The generated 3D scenes are significantly sharper and more complete than those produced by state-of-the-art baselines. Crucially, the model can generate diverse samples for occluded or ambiguous areas, providing multiple plausible interpretations of the unseen parts, all while maintaining multi-view consistency.
This research marks a significant step towards creating complete, high-fidelity 3D models from minimal input, opening new possibilities for applications in augmented reality, virtual worlds, and robotics.
