TLDR: HoloGarment is a novel method that generates photorealistic 360-degree views of real-world garments from just 1-3 images or a video. It uses an implicit training paradigm, combining large-scale real 2D video data with smaller synthetic 3D data to overcome challenges like occlusions and deformations. The system also introduces a “garment atlas” for enhanced video-to-360° synthesis, achieving state-of-the-art performance in digital garment visualization for applications like online retail and virtual try-on.
Researchers from the University of Washington and Google have introduced HoloGarment, a groundbreaking method designed to generate complete 360-degree novel views of real-world garments from limited input like a few images or a video. This innovation addresses significant challenges in digital fashion, such as complex human poses, occlusions, and cloth deformations, which have historically hindered the creation of high-quality digital garment visualizations.
Traditional methods for novel view synthesis often rely heavily on synthetic 3D training data, which typically features unoccluded and static objects. This reliance leads to poor generalization when applied to the diverse and dynamic nature of real-world clothing. HoloGarment tackles this by bridging the gap between real and synthetic data through a novel implicit training paradigm.
The Core Innovation: Implicit Training
HoloGarment’s key insight lies in its implicit training strategy. Instead of directly training on paired real-world 3D data (which is scarce), the model learns from two distinct but related tasks:
1. Garment Animation: Using large-scale real 2D image and video data, the model learns to handle the complexities of real-world garments, including wrinkles, occlusions, and pose variations.
2. Novel View Synthesis: Utilizing smaller-scale synthetic 3D data, the model learns to generate unoccluded, static 360-degree views of garments in a canonical pose.
By optimizing a shared garment embedding space between these two domains, HoloGarment implicitly learns to perform 3D-consistent novel view synthesis from real-world inputs. This means it can take an image of a person wearing a garment in a dynamic, possibly occluded pose, and generate a photorealistic 360-degree spin of that garment in a clean, static A-pose.
Garment “Atlas” for Video-to-360° Synthesis
Further enhancing its capabilities, HoloGarment introduces the concept of a “garment atlas.” This is a finetuned garment-specific embedding optimized on a dynamic video of a person wearing the garment. The atlas effectively consolidates information from an arbitrary number of video frames, including various views, poses, and deformations, into a unified 360-degree garment representation. This eliminates the need for careful input view selection and significantly improves the fidelity and realism of the synthesized views from video inputs.
Architecture and Performance
At its heart, HoloGarment is an image- and pose-conditioned video diffusion model. It builds upon existing video transformer diffusion models but incorporates disjoint temporal blocks to separately handle dynamic video motion and static 3D spin motion. This architectural choice allows the model to better disentangle and synthesize different motion styles.
Extensive experiments demonstrate that HoloGarment achieves state-of-the-art performance in novel view synthesis of in-the-wild garments from both images and videos. It robustly handles challenging real-world artifacts while maintaining photorealism, view consistency, fine texture details, and accurate geometry. The method has been shown to outperform other state-of-the-art techniques across various metrics and datasets.
Also Read:
- A New Approach to One-Step Image Super-Resolution: Balancing Detail and Realism
- SignClip: A New Framework for Accurate Sign Language Translation Using Gestures and Mouthing
Applications and Future Directions
The demand for high-quality digital garment visualizations is rapidly growing, driven by online retail, virtual try-on experiences, and digital fashion design. HoloGarment offers a powerful tool to meet this demand by providing full 360-degree representations of garments that are independent of the wearer and free from occlusions or wrinkles.
While HoloGarment represents a significant leap forward, the researchers acknowledge certain limitations. These include challenges with unusual garment shapes (e.g., asymmetry or cut-outs) due to the diversity limitations of synthetic 3D datasets, and a bias towards garment categories more abundant in the 3D data. Future work aims to address these by incorporating larger synthetic garment datasets, speeding up the atlas finetuning process, and increasing resolution via super-resolution networks.
For more in-depth technical details, you can read the full research paper here.
