TLDR: MeshSplat is a new framework that reconstructs detailed 3D surfaces from sparse input images. It utilizes 2D Gaussian Splatting (2DGS) as a key representation, which is well-suited for thin surfaces. The method employs a feed-forward network, a Weighted Chamfer Distance Loss for accurate point positioning, and an uncertainty-guided normal prediction network for precise surface orientation. Experiments show MeshSplat achieves superior performance in generalizable sparse-view mesh reconstruction and cross-dataset generalization compared to existing methods.
Researchers have introduced MeshSplat, a novel framework designed to tackle the challenging problem of reconstructing accurate 3D surfaces from a limited number of input images. This new approach leverages 2D Gaussian Splatting (2DGS) to achieve state-of-the-art performance in generalizable sparse-view surface reconstruction, a task crucial for applications in augmented reality, virtual reality, and robotics.
Traditional methods for 3D surface reconstruction often struggle when only a few images are available, as sparse views provide insufficient geometric information for high-quality optimization. MeshSplat addresses this by creating a feed-forward system that learns geometric priors from diverse scenarios, enabling it to generalize across different scenes.
The core innovation of MeshSplat lies in its use of 2DGS as a bridge. Unlike 3D Gaussian Splatting (3DGS), which uses ellipsoid shapes that can lead to inconsistent surfaces, 2DGS is naturally better suited for representing thin surfaces and can be directly used for mesh extraction. This allows MeshSplat to learn generalizable geometric information from the task of synthesizing new views, even without direct 3D ground-truth supervision.
To ensure the accuracy of the 2DGS, MeshSplat incorporates two key designs. First, a Weighted Chamfer Distance Loss is introduced to improve the precision of 2DGS position prediction. This loss function aligns point clouds generated from predicted depths across different views, giving higher importance to overlapping regions where matching confidence is strong. Second, an uncertainty-based normal prediction network is used to enhance the accuracy of 2DGS orientation. This network predicts per-view surface normals, which are then converted into rotation quaternions for the 2DGS, ensuring they align precisely with the actual scene surfaces. This is supervised by an existing monocular normal estimator, which provides valuable geometric priors.
The MeshSplat framework takes sparse images as input and processes them through a multi-view backbone to extract features. These features are then used to construct cost volumes, which help in generating coarse depth maps. A Gaussian Prediction Network, comprising a depth refinement network and the normal prediction network, then derives the precise attributes for the pixel-aligned 2DGS. These 2DGS can then be used to render novel views for supervision and, crucially, to reconstruct the scene mesh.
Extensive experiments conducted on datasets like Re10K and Scannet demonstrate that MeshSplat significantly outperforms existing methods in terms of mesh reconstruction quality. The reconstructed meshes are smoother and more complete, even for complex scenes with sparse input views. Furthermore, MeshSplat shows strong cross-dataset generalization capabilities, performing well on unseen datasets like Replica when trained on Re10K.
While MeshSplat marks a significant advancement, the researchers acknowledge some limitations. The model may occasionally predict discontinuous depth maps in areas with weak textures due to feature matching ambiguities. Additionally, it cannot reconstruct surfaces in regions not visible from the input views, suggesting future work could explore generative approaches to address this.
Also Read:
- HOSt3R: Capturing Detailed 3D Hand-Object Interactions from Standard Images
- Guiding Robots with Spatial-Aware Vision and Action
In conclusion, MeshSplat offers an end-to-end, generalizable solution for high-quality 3D surface reconstruction from sparse images, paving the way for more accurate and efficient 3D modeling in various applications. For more details, you can read the full research paper here.
