TLDR: LuxDiT is a new AI model that uses a video diffusion transformer to accurately estimate high-dynamic-range (HDR) scene lighting from single images or videos. It addresses challenges like data scarcity and indirect visual cues by training on a large synthetic dataset and fine-tuning with real HDR panoramas using a technique called LoRA. The model generates detailed and scene-consistent HDR environment maps, outperforming previous methods in various evaluations and enabling more realistic virtual object insertion.
Estimating how a scene is lit from a single image or video has long been a complex challenge in computer vision and graphics. This task is particularly difficult because it relies on subtle visual cues, requires understanding the global context of a scene, and needs to produce high-dynamic-range (HDR) outputs, which capture a much wider range of light intensities than standard images. Traditional learning methods often struggle due to the scarcity of high-quality, ground-truth HDR environment maps, which are expensive and difficult to capture.
A new approach, called LuxDiT, tackles this problem by fine-tuning a video diffusion transformer to generate HDR environment maps conditioned on visual input. This novel data-driven model learns to infer illumination from indirect visual cues and shows impressive generalization to real-world scenes, even after being trained on a large synthetic dataset with diverse lighting conditions.
How LuxDiT Works
LuxDiT formulates lighting estimation as a conditional generative task. It takes an input image or video and synthesizes HDR panoramas from noise using a diffusion transformer (DiT). DiTs are well-suited for this task because their attention-based architecture can aggregate global context, helping the model reason from indirect cues like shading and reflections.
To handle the challenge of representing HDR content, which standard image models aren’t designed for, LuxDiT uses a dual-tonemapping strategy. It represents each HDR panorama using two complementary low-dynamic-range (LDR) versions: one with standard Reinhard tonemapping (Eldr) and another capturing normalized log-intensity (Elog). These are then encoded into a latent space for the diffusion model to process. At inference time, a lightweight neural network reconstructs the final HDR panorama from these two representations.
For conditioning the model on visual input, LuxDiT encodes the input video into a sequence of tokens. This attention-based approach allows the model to extract fine-grained shading cues like shadow orientation, surface reflections, and specular highlights, which are crucial for accurate lighting estimation.
Training Strategy and Data
Training such a model requires extensive and diverse lighting data. To overcome the lack of real-world HDR lighting supervision, LuxDiT uses a multi-pronged data strategy:
-
Synthetic Rendering Data: A large-scale synthetic dataset is created by rendering randomized 3D scenes with varied geometry, materials, and lighting conditions. This data is essential for teaching the model physically accurate shading cues and light-source locations.
-
HDR Panorama Images: Training pairs are generated by sampling perspective crops from real HDR environment maps, with random camera parameters and exposure settings. This helps the model learn to predict full panoramas from limited views.
-
LDR Panorama Videos: To improve robustness and temporal consistency for video inputs, LDR panoramic videos are also incorporated. While these lack HDR ground truth, they expose the model to natural image statistics and motion patterns.
LuxDiT employs a two-stage training strategy. The first stage focuses on learning fundamental lighting cues from the synthetic data. The second stage then adapts the model to real-world distributions using a technique called Low-Rank Adaptation (LoRA) fine-tuning on curated real HDR panoramas. LoRA helps improve the semantic alignment between the input scene and the predicted environment map without overfitting or losing the model’s initial capacity.
Also Read:
- Generating Immersive Scene Views: A Two-Stage Approach for Consistent 360-Degree Videos
- LatentEdit: A New Approach to Consistent Image Editing with Diffusion Models
Performance and Applications
LuxDiT has been rigorously evaluated on several benchmark datasets, including Laval Indoor, Laval Outdoor, and Poly Haven. It consistently outperforms existing state-of-the-art methods in both quantitative and qualitative assessments. For instance, it significantly reduces lighting estimation error and improves temporal consistency for video inputs. On outdoor scenes with concentrated sunlight, LuxDiT reduces peak angular error by nearly 50% compared to previous methods, which is critical for casting realistic shadows in applications like virtual object insertion.
A key application of LuxDiT is realistic virtual object insertion. The model produces high-quality HDR environment maps that enable virtual objects to be seamlessly integrated into real scenes with accurate shadows and reflections. User studies confirm that LuxDiT achieves visual quality comparable to specialized object insertion methods, even though LuxDiT focuses solely on lighting estimation.
While LuxDiT represents a significant step forward, its iterative nature means inference can be computationally intensive, limiting real-time applications. Future work aims to explore more efficient architectures or model distillation techniques to accelerate inference. Additionally, generating higher-resolution panoramas will require richer and more diverse HDR supervision. This research, detailed further in the paper LuxDiT: Lighting Estimation with Video Diffusion Transformer, paves the way for more unified inverse and forward rendering frameworks, potentially leading to advancements in general-purpose scene reconstruction and appearance synthesis.
