TLDR: DirectGaussian is a novel text-to-3D generation method that creates high-fidelity 3D object surfaces using 2D Gaussian Splatting. It employs multi-view normal and texture priors, guided by text, to generate initial coarse Gaussian surfels. A key innovation is the introduction of 360-degree surround-view surface curvature and convergence constraints during optimization, which ensures geometric consistency and fine detail preservation across all viewpoints, effectively addressing common issues like inconsistent geometry in existing methods.
The field of Text-to-3D modeling is rapidly advancing, offering exciting possibilities for creating three-dimensional content from simple text descriptions. However, generating complex and realistic 3D objects remains a significant challenge due to the intricate geometric shapes found in the real world. Existing methods often struggle with issues like discrete visual outputs from point clouds, high rendering costs associated with neural radiance fields (NeRFs), or limitations imposed by predefined topologies in mesh-based approaches.
A new research paper, titled “Generating Surface for Text-to-3D using 2D Gaussian Splatting,” introduces a novel method called DirectGaussian. This approach tackles the challenge by focusing on generating the surfaces of 3D objects, represented by what are known as surfels (surface elements). DirectGaussian leverages conditional text generation models and renders the 3D object’s surface using 2D Gaussian splatting, incorporating multi-view normal and texture information. A key innovation is the inclusion of curvature constraints during the optimization process, which helps maintain geometric consistency across different viewpoints.
Traditional 3D Gaussian Splatting (3DGS) has shown great promise for high-fidelity rendering and fast training, but it typically requires an initial point cloud, often reconstructed using Structure-from-Motion (SfM). This dependency can lead to performance drops if the initial point cloud is noisy or randomly initialized. Furthermore, volumetric 3D Gaussians can struggle to capture the fine details of thin surfaces. DirectGaussian addresses these limitations by building upon 2D Gaussian Splatting, which uses 2D Gaussian primitives to represent 3D scenes, offering more accurate geometric representations.
The core idea behind DirectGaussian is to generate Gaussian surfels that inherently encode multi-view 2D information in a geometrically consistent manner. This provides high-quality initial inputs for the splatting process, which is crucial for achieving accurate geometry and texture. Instead of relying on SfM, DirectGaussian uses multi-view normal and texture priors to align initial Gaussians with corresponding image conditions, simplifying parameter optimization.
The framework consists of three main components: first, constructing a dataset of Gaussian surfels aligned with textual descriptions; second, a generative model for creating coarse Gaussian surfels; and third, an optimization process guided by multi-view priors using splatting loss and novel surface constraints.
To overcome the limitations of optimizing with a small number of viewpoints, DirectGaussian introduces two critical constraints: a 360° surround-view surface curvature constraint and surface convergence constraints. The curvature constraint ensures that the surface geometry remains consistent when viewed from different angles, effectively preventing distortions. The surface convergence constraint encourages surfels to stay close to the actual object surface, leading to smoother geometry and more consistent textures.
The process begins by generating coarse Gaussian surfel parameters (position, color, opacity, scale, and rotation) using a multi-head attention model. This model is guided by normal and texture information derived from text-conditioned multi-view diffusion models. These coarse Gaussians are then refined through an optimization loop that minimizes a comprehensive loss function, including the standard splatting loss (for RGB texture, depth distortion, and normal consistency) along with the newly introduced curvature and convergence losses.
Experiments demonstrate that DirectGaussian is capable of creating diverse and high-fidelity 3D content. The method produces textured 3D objects that align well with text descriptions and exhibit improved overall consistency and authenticity, especially when viewed from novel perspectives. It effectively mitigates the “Multi-face Janus problem,” where objects appear inconsistent or distorted from unseen angles, a common issue in other multi-view reconstruction methods. Quantitative evaluations using the CLIP score also show that DirectGaussian achieves better results compared to several baseline methods, indicating superior generation quality and relevance to text prompts.
The researchers also conducted further studies to highlight the importance of their coarse Gaussian generation model and the surface curvature constraint. They found that using their learned coarse Gaussians leads to more stable convergence and better geometric fidelity compared to random initialization. Similarly, the curvature loss significantly contributes to smoother surface transitions and improved consistency across neighboring viewpoints.
Also Read:
- Guided Learned Vertex Descent: A New Era for 3D Face Reconstruction
- PointNSP: Advancing 3D Point Cloud Generation with Multi-Scale Prediction
In conclusion, DirectGaussian presents an efficient and robust framework for text-to-3D generation. By combining anisotropic surfels with 2D Gaussian Splatting and incorporating innovative 360-degree curvature constraints, the method generates surfaces with realistic lighting effects and diverse geometric structures. This advancement holds promising application prospects in various fields, including animation, virtual reality, and video game content creation. You can read the full paper here: Generating Surface for Text-to-3D using 2D Gaussian Splatting.
