TLDR: WeatherDiffusion is a new AI framework that improves how autonomous driving systems understand and reconstruct scenes, especially in challenging weather. It uses a weather-guided diffusion model and a novel attention mechanism (MAA) to accurately estimate scene properties like material, geometry, and lighting from images. The model can also edit weather and lighting conditions in rendered images. Supported by two new datasets (WeatherSynthetic and WeatherReal), WeatherDiffusion significantly boosts the performance of downstream tasks like object detection and image segmentation in adverse weather, making autonomous vehicles more robust.
A new research paper introduces WeatherDiffusion, a groundbreaking framework designed to enhance how autonomous driving (AD) systems perceive and interact with their environment, especially under challenging weather conditions. This innovative model tackles the complexities of forward and inverse rendering, two critical techniques for understanding and reconstructing real-world scenes.
Forward rendering involves generating realistic images under various lighting and weather conditions, which helps train AI models to better understand scenes. Inverse rendering, on the other hand, aims to recover the fundamental properties of a scene—such as its geometry, material, and lighting—from observed images. This allows for more flexible applications like editing materials or relighting a scene.
Traditional methods for these tasks often struggle with the dynamic and unpredictable nature of autonomous driving scenarios, particularly when faced with rain, fog, or snow. These conditions drastically alter lighting, obscure geometric details, and affect surface appearances, making accurate scene understanding difficult. While recent advancements in large diffusion models have shown promise, they often lack the control and robustness needed for real-world AD applications.
Introducing WeatherDiffusion
WeatherDiffusion addresses these challenges by leveraging a sophisticated diffusion-based framework. It enables the authentic estimation of material properties, scene geometry, and lighting from images. Crucially, it also supports controllable editing of weather and illumination, guided by text descriptions and predicted intrinsic maps—detailed representations of a scene’s underlying properties.
The model is built upon a fine-tuned version of Stable Diffusion 3.5, a powerful generative AI model. To make it robust against diverse weather, the researchers introduced a ‘weather controller’ that categorizes different weather conditions, helping the model distinguish between subtle visual cues, such as low visibility versus shadowed regions.
Intrinsic Map-Aware Attention (MAA)
A key innovation in WeatherDiffusion is the Intrinsic Map-Aware Attention (MAA) module. This component replaces traditional text-based guidance with visual cues, allowing the model to focus on specific regions of an image that are most relevant to the intrinsic map being generated. For instance, when predicting the ‘albedo’ (the true color of an object without lighting effects), MAA directs the model’s attention to fine-grained object textures. For ‘metallicity’, it focuses on metallic objects like cars or poles. This targeted attention significantly improves the quality of inverse rendering.
New Datasets for Real-World Scenarios
To train and validate WeatherDiffusion, the researchers created two new datasets: WeatherSynthetic and WeatherReal. WeatherSynthetic is a large-scale synthetic dataset comprising 35,000 images rendered in Unreal Engine 5, covering a wide range of weather types (sunny, rainy, snowy, foggy, etc.), times of day, and environments (urban, suburban, highway). WeatherReal is a 10,000-image real-world dataset, created by applying weather augmentation techniques to existing open-source datasets like Waymo and Kitti. These datasets are crucial for bridging the gap between synthetic training data and complex real-world conditions.
Also Read:
- New AI Model Predicts Pedestrian Paths by Understanding Their Intentions
- Unlocking 3D Surface Normals from Single Images with a Self-Supervised Framework
Impact on Autonomous Driving
Extensive experiments show that WeatherDiffusion outperforms existing state-of-the-art methods in both forward and inverse rendering tasks. Beyond its core capabilities, the model demonstrates significant value for downstream tasks in autonomous driving, such as object detection and image segmentation. By effectively removing environmental distortions like heavy snow or fog from visual inputs, WeatherDiffusion provides cleaner images that allow detection and segmentation models to perform more accurately, even in adverse weather scenarios.
While WeatherDiffusion represents a major step forward, the researchers acknowledge some limitations, such as challenges with out-of-distribution objects (e.g., cranes not seen during training) and distinguishing between sky and buildings in extremely dense fog. Future work aims to reduce dependence on high-quality training data and explore combining diffusion models with reinforcement learning.
For more technical details, you can read the full research paper here: WeatherDiffusion: Weather-Guided Diffusion Model for Forward and Inverse Rendering.
