TLDR: SeqTex is a novel end-to-end framework that uses pre-trained video foundation models to directly generate high-quality UV texture maps for 3D meshes. It redefines texture generation as a sequence prediction problem, combining multi-view image synthesis with UV texture mapping. This approach leads to superior 3D consistency, texture-geometry alignment, and visual fidelity compared to previous methods, effectively addressing challenges like data scarcity and spatial inconsistencies in 3D texture generation.
Creating realistic textures for 3D models has always been a time-consuming and challenging task for artists. Traditional methods often involve manual effort or multi-stage digital processes that can lead to errors and inconsistencies across the 3D surface. This challenge is particularly significant in industries like gaming and film, where thousands of high-quality textured models are needed.
Despite rapid advancements in generative AI for images and videos, 3D texture generation has lagged. A major hurdle is the lack of large, high-quality 3D texture datasets. Existing approaches often fine-tune image generative models, but these typically produce only multi-view images, requiring additional steps to create the essential UV texture maps used in modern graphics pipelines. These multi-stage pipelines are prone to accumulating errors and creating spatial inconsistencies on the 3D surface.
Introducing SeqTex: A Breakthrough in 3D Texture Generation
A new research paper introduces SeqTex, a novel end-to-end framework designed to overcome these limitations. SeqTex leverages the vast visual knowledge embedded in pre-trained video foundation models to directly generate complete UV texture maps. Unlike previous methods that treat UV textures in isolation, SeqTex redefines the problem as a sequence generation task. This allows the model to learn the combined distribution of multi-view renderings and UV textures, effectively transferring consistent image-space knowledge from video models into the UV domain.
How SeqTex Works
SeqTex takes an untextured 3D mesh and, optionally, an image or text input. It then uses a pre-trained video diffusion model to simultaneously synthesize multi-view images of the object and its UV texture map. This joint prediction is treated as a “video” sequence, where the UV texture map is the final frame. This approach offers several key advantages:
- It aligns the task with the temporal structure of video foundation models, transferring learned visual knowledge to textures.
- By incorporating multi-view context, it integrates information from different viewpoints for more coherent and realistic UV textures.
- The unified architecture allows training with additional high-quality multi-view-only datasets, enhancing generalization.
Key Innovations
The SeqTex architecture introduces several innovations:
-
Decoupled Multi-View (MV) and UV Texture Learning: To bridge the gap between spatially continuous multi-view images and the often discontinuous UV map layout, SeqTex uses separate processing branches. The MV branch efficiently adapts video priors using a lightweight fine-tuning method (LoRA), while the UV branch is fully fine-tuned for high-fidelity texture maps.
-
Geometry-Informed Attention: This mechanism uses 3D geometric information, such as global positions and normals, to guide the model. It helps UV tokens focus on relevant regions in multi-view tokens that correspond to the same 3D locations, ensuring precise alignment between the image and UV domains.
-
Adaptive Token Resolution: To capture fine texture details without excessive computational cost, UV textures are processed at a higher resolution (1024×1024 pixels), while multi-view images are generated at a lower resolution (512×512 pixels).
Training and Performance
SeqTex employs a multi-task learning strategy, supporting both image-to-texture and geometry-to-multi-view tasks. For image-to-texture generation, it uses lighting-free albedo maps to ensure consistency. For geometry-to-multi-view, it uses illuminated images, which are more compatible with natural video data and allow for broader dataset integration.
Extensive experiments show that SeqTex achieves state-of-the-art performance in both image-conditioned and text-conditioned 3D texture generation. It consistently surpasses previous methods in terms of 3D consistency, texture-geometry alignment, and visual fidelity, while maintaining competitive processing speeds. Ablation studies further confirm the critical role of video priors, joint multi-view and UV modeling, and the decoupled branch design in achieving these superior results.
Also Read:
- 3DGeoMeshNet: A Multi-Scale Graph Neural Network for High-Fidelity 3D Mesh Reconstruction
- LighthouseGS: Bringing High-Quality 3D Scene Reconstruction to Your Mobile Phone
Conclusion
SeqTex represents a significant step forward in 3D content creation. By effectively adapting pre-trained video foundation models for end-to-end UV texture map generation, it addresses long-standing challenges related to data scarcity and UV spatial discontinuity. This framework establishes a strong foundation for integrating advanced vision models into practical 3D pipelines, opening new possibilities for scalable and robust texture synthesis.
