TLDR: Researchers from Columbia University have developed a new method for estimating spatiotemporally consistent indoor lighting from videos, even when lighting conditions change dynamically. Their approach uses 2D diffusion models to ‘inpaint’ virtual chrome balls as light probes, training a neural network (MLP) to represent a continuous light field. This enables highly realistic virtual object insertion in augmented reality and video composition, outperforming previous methods in consistency and detail.
Estimating realistic lighting conditions within indoor environments, especially from videos where lighting changes over time and across different locations, has long been a significant challenge in computer graphics and vision. This difficulty arises because the process is inherently complex; the estimated lighting needs to be highly detailed and capture light from all directions, even when the input images are of lower quality or limited in scope. Existing methods often fall short, either by only predicting a single, global lighting condition for an entire scene or by struggling to maintain consistency when lighting varies dynamically.
A new research paper titled “Spatiotemporally Consistent Indoor Lighting Estimation with Diffusion Priors” introduces a novel approach to tackle this problem. Authored by Mutian Tong, Rundi Wu, and Changxi Zheng from Columbia University, their method can estimate a continuous light field from an input video. This light field accurately describes how illumination changes both spatially (as you move around the scene) and temporally (as time passes and lights might turn on or off).
The core of their innovation lies in leveraging advanced 2D diffusion models, a type of artificial intelligence model known for generating realistic images. They represent the scene’s lighting as a multi-layer perceptron (MLP), which is a type of neural network. To train this system, they fine-tune a pre-trained image diffusion model to predict lighting at multiple locations simultaneously. This is achieved by a clever technique: the model learns to “inpaint” multiple virtual chrome balls into an image, treating these balls as light probes that reflect the surrounding environment. By jointly processing these virtual probes, the system can infer consistent lighting across different points in the scene.
Unlike previous methods that might only estimate lighting at a single viewpoint or struggle with dynamic changes, this new approach is designed for “in-the-wild” videos, meaning real-world footage with unpredictable lighting variations. The researchers built a synthetic dataset using a procedural indoor scene generator called Infinigen Indoors to train their model, allowing them to render ground-truth lighting information at various spatial locations.
The method then distills this learned knowledge from the 2D diffusion model into the MLP-represented light field. This process iteratively refines the MLP by comparing its rendered chrome balls under the estimated lighting with the image priors learned by the diffusion model. This ensures that the estimated lighting is not only accurate but also visually consistent across frames and locations.
Also Read:
- Achieving Realistic Real-Time Lighting in Virtual Worlds
- Neural Vertex Features: A Memory-Efficient Approach for Global Illumination in 3D Rendering
The paper demonstrates superior performance over existing baselines in indoor lighting estimation from both single images and videos. This breakthrough is particularly significant for applications like augmented reality and video composition, where high-quality, consistent lighting is crucial for seamlessly inserting virtual objects into real-world footage. The ability to estimate spatiotemporally consistent lighting from dynamic videos is a rarely achieved feat in prior works, making this research a notable advancement in the field. For more details, you can read the full paper here.
