TLDR: This research introduces two novel uncertainty-aware training approaches to improve the Segment Anything Model (SAM and SAM2) for self-driving cars in adverse weather. One method involves finetuning SAM2 with an uncertainty-aware loss function for overall scene segmentation, showing significant accuracy gains. The second adapts an Uncertainty-Aware Adapter (UAT) to SAM, specifically enhancing object instance segmentation in extreme weather by focusing on ambiguous regions. Both approaches demonstrate improved robustness and reliability, highlighting the importance of uncertainty quantification for safer autonomous driving.
Self-driving cars hold immense promise for the future of transportation, but their safe operation hinges on robust perception systems, especially in challenging environmental conditions. Adverse weather, such as rain, snow, and fog, significantly degrades image quality, making it difficult for autonomous vehicles to accurately perceive their surroundings. Traditional image segmentation models, including advanced vision foundation models like Segment Anything Model (SAM) and its successor SAM2, often struggle in these scenarios due to their inability to quantify uncertainty in their predictions.
Drawing inspiration from successes in medical imaging, where uncertainty-aware training has proven effective in handling ambiguous cases, new research from Queen’s University explores two innovative approaches to enhance segmentation performance for self-driving systems in adverse weather. The core idea is to explicitly incorporate uncertainty modeling into the training process, allowing the models to better understand and adapt to visual ambiguities.
Two Paths to Enhanced Perception
The researchers propose a dual strategy to tackle the problem:
- Multi-step Finetuning for SAM2: This approach focuses on improving overall scene recognition across varying weather conditions. It involves a detailed finetuning process for SAM2, integrating uncertainty metrics directly into the loss function during training.
- Uncertainty-Aware Adapter (UAT) for SAM: This method is designed to address specific object instance segmentations, particularly in extreme weather scenarios. It adapts a novel Uncertainty-Aware Adapter, originally developed for medical image segmentation, to the context of autonomous driving.
Finetuning SAM2 for Broader Accuracy
For the first approach, the team developed a multi-step finetuning process for SAM2 using driving datasets like BDD100K and CamVid. The key innovation lies in a custom loss function that combines three components:
- Binary Cross-Entropy Loss: Helps the model learn underlying distributions in driving images.
- Intersection-over-Union (IoU) Loss: Penalizes incorrect segmentations, ensuring objects are neither missed nor over-segmented.
- Monte Carlo Uncertainty Loss: This crucial component involves feeding the input image multiple times to the model to generate several mask predictions. The pixel-wise standard deviation of these predictions creates an uncertainty tensor, which is then used to weight the final loss. This directs the model to reduce uncertainty and variability in its outputs, especially in ambiguous regions.
The finetuned SAM2 model demonstrated significant improvements over the zero-shot SAM2, with an average IoU increase of 36.13% and DICE score improvement of 48.79%. It showed strong gains in segmenting common road objects like cars and people, and importantly, it generalized well across diverse real-world and synthetic driving environments, including BDD100K, CamVid, and GTA5 Driving Dataset. A notable benefit was its ability to consistently assign a single, accurate mask per vehicle, reducing fragmented outputs often seen in zero-shot models.
UAT Adapter for Extreme Conditions
The second approach leverages the Uncertainty-Aware Adapter (UAT) architecture with the original SAM model, specifically targeting object segmentation in severe weather. The UAT adapter, inspired by medical imaging techniques, is a lightweight component inserted into each transformer block of SAM. It uses a Conditional Variational Autoencoder (CV AE) to incorporate a sampled uncertainty code, which is then integrated with the main features through learnable attention-like mechanisms.
To train this model, the CamVid dataset underwent extensive preprocessing. Original images were subjected to random fog, rain, or snow filters to simulate adverse conditions. Crucially, to meet the UAT adapter’s requirement for multiple ground truth segmentations (common in ambiguous medical images), elastic deformations were applied to the single human-segmented ground truths. This generated three additional plausible interpretations for each image, mimicking the inherent ambiguity of real-world adverse weather. The training specifically focused on car instance segmentations, allowing the model to prioritize regions with high variability.
The UAT-SAM showed remarkable resilience in heavily obscured conditions where zero-shot SAM often failed completely. It achieved a 30% increase in the DICE coefficient and a 42.7% increase in IoU scores compared to zero-shot SAM for car instance segmentations in heavy weather. This highlights its enhanced ability to handle complex segmentation tasks by focusing on regions of high uncertainty, leading to more reliable segmentation even when visibility is severely compromised.
Also Read:
- Knowledge-Driven AI Policy Enhances End-to-End Autonomous Driving
- ADClick: Accelerating Anomaly Detection with Smart Labeling
A Safer Future for Autonomous Driving
This research underscores the critical importance of explicit uncertainty modeling in safety-critical autonomous driving applications. The two complementary approaches address different facets of the inclement weather challenge: the UAT adapter provides a targeted solution for the most severe conditions where immediate safety decisions are paramount, while the uncertainty-finetuned SAM2 offers broader improvements in overall segmentation quality across varying weather. These advancements not only push the boundaries of semantic segmentation in challenging conditions but also demonstrate the significant value of integrating uncertainty awareness into modern foundation models like SAM and SAM2. For more details, you can read the full research paper here.
