TLDR: RoSe is a novel self-supervised stereo matching method designed to perform robustly under adverse weather conditions (night, rain, fog). It addresses key weaknesses of existing methods by integrating visual foundation models for enhanced feature extraction and introducing an Anti-Adverse Feature Enhancement Module. Furthermore, it employs a two-step training paradigm: self-supervised scene correspondence learning using synthetic paired datasets and adverse weather distillation with pseudo-labels. This approach significantly outperforms current state-of-the-art self-supervised methods, demonstrating improved accuracy and reliability in diverse and challenging scenarios.
Disparity estimation, a crucial task for autonomous driving and reconstructing 3D scenes, involves determining the depth of objects from stereo images. While traditional deep learning methods have shown impressive results, they often require extensive, costly ground truth data. Self-supervised methods, which learn from the data itself without explicit labels, offer a promising alternative, allowing for massive scalability. However, these methods typically struggle significantly under challenging environmental conditions such as night, rain, and fog.
A new research paper titled “RoSe: Robust Self-supervised Stereo Matching under Adverse Weather Conditions” by Yun Wang, Junjie Hu, Junhui Hou, Chenghao Zhang, Renwei Yang, and Dapeng Oliver Wu, introduces a novel approach to overcome these limitations. The authors identify two primary reasons for the performance drop in existing self-supervised stereo matching methods during adverse weather: first, the inability of standard CNN-based feature extractors to cope with noise and reduced visibility in degraded areas like reflective or textureless surfaces; and second, the disruption of accurate pixel correspondences, which undermines the photometric consistency assumption that many self-supervised methods rely on for supervision.
To address the first challenge, RoSe proposes a robust feature extractor. It enhances traditional CNN-based Feature Pyramid Networks (FPNs) by incorporating powerful priors from visual foundation models (VFMs) like Segment Anything (SAM) and Depth Anything V2 (DAMv2). These VFMs are known for their strong performance in dense prediction tasks. To further improve robustness, RoSe introduces an Anti-Adverse Feature Enhancement Module (AFEM). This module processes features across spatial, channel, and frequency domains, effectively separating weather-related noise from essential scene information. This ensures that the extracted features remain consistent and reliable, even in challenging conditions, much like features derived from clear images.
For the second challenge of ineffective supervision, RoSe moves beyond solely relying on the photometric consistency assumption. The core idea is to leverage a “scene correspondence prior,” which states that a clear image pair and its corresponding adverse weather pair should share the same semantic context and depth information. Since collecting real-world paired datasets under diverse weather conditions is difficult, the researchers generated synthetic stereo datasets. They used an advanced image-to-image translation model (CycleGAN-Turbo) to create realistic adverse weather versions (rainy, foggy, night) from clear stereo pairs, ensuring that the semantic context and disparity remained consistent.
This knowledge of scene correspondence forms the basis of RoSe’s two-step self-supervised training paradigm:
1. Self-supervised Scene Correspondence Learning
In this initial step, the model processes clear and adverse image pairs through two separate branches. By exploiting the scene correspondence prior, two new consistency losses are introduced: a Feature Consistency Loss and a Disparity Consistency Loss. These losses ensure that the learned features and disparity values are consistent between clear and adverse weather conditions, guiding the model to learn a latent space that is less affected by weather degradations. This step provides robust supervisory signals, preventing the model from collapsing, which can happen if only consistency losses are used.
Also Read:
- Advancing 2D-3D Sensor Alignment for Autonomous Systems with CrossI2P
- Achieving Clearer Object Edges in AI Depth Perception
2. Adverse Weather Distillation
The second step addresses the remaining weaknesses of the photometric consistency assumption, particularly in difficult areas like occlusions. A “teacher” model, which is the frozen model trained in Step 1, generates high-quality pseudo disparity maps from clear image pairs. These pseudo labels are then used to supervise a “student” model. The student model is trained with mixed inputs (clear and adverse) and augmented with techniques like asymmetric occlusions, simulating photometric consistency failures. This distillation process teaches the student model to infer accurate disparity maps even under challenging conditions, significantly enhancing its reliability and performance.
Extensive experiments on datasets like DrivingStereo, MS2, and KITTI demonstrate the effectiveness of RoSe. The method significantly outperforms existing state-of-the-art self-supervised stereo matching solutions across various adverse weather conditions, including night, rain, and fog. It also shows strong performance on standard clear weather benchmarks like KITTI, highlighting its versatility. The authors note that RoSe achieves competitive results even with a more modest architecture compared to some domain-generalized methods, and it requires significantly fewer training samples than other state-of-the-art self-supervised methods on KITTI.
While RoSe marks a significant advancement, the authors acknowledge limitations. Its performance heavily relies on the realism and consistency of the synthetic adverse weather pairs generated by image-to-image translation models. Inaccuracies in these translations could introduce noise. Furthermore, under extremely severe image degradation, RoSe may still produce unreliable disparity estimates, which is a critical concern for safety-sensitive applications like autonomous driving.
In conclusion, RoSe offers a robust and effective solution to the long-standing problem of self-supervised stereo matching under adverse weather conditions. By combining insights from visual foundation models with a novel two-step training approach, it significantly improves depth estimation, contributing to safer and more reliable autonomous systems in diverse environments.
