TLDR: A new research paper introduces WiSE-OD, a method that improves the robustness of infrared object detection models against various image corruptions and cross-modality shifts. By ensembling the weights of RGB-pretrained and IR fine-tuned models, WiSE-OD significantly boosts performance without additional training or inference costs, and also introduces new corrupted IR benchmarks (LLVIP-C and FLIR-C) for evaluation.
Object detection in infrared (IR) imagery is crucial for applications in low-light and nighttime conditions, such as surveillance and autonomous driving. Unlike standard cameras that rely on visible light (RGB), IR cameras capture heat emitted by objects, making them effective regardless of lighting. However, a significant challenge in IR object detection is the scarcity of large-scale IR datasets. This often forces developers to use models initially trained on vast RGB image datasets, like COCO, and then fine-tune them for IR specific tasks.
While fine-tuning these RGB-pretrained models on IR data can improve their accuracy within the specific IR domain, it frequently compromises their ability to perform well when faced with unexpected variations or “distribution shifts” in the input data. This is known as a lack of robustness. The inherent differences between RGB and IR modalities further complicate this transfer learning process, making it difficult for models to generalize to new or diverse scenarios.
To address these critical issues, researchers have introduced two new benchmarks: LLVIP-C and FLIR-C. These benchmarks are created by applying various common corruptions, such as noise, blur, and changes in brightness or contrast, to existing standard IR datasets (LLVIP and FLIR). These corrupted datasets provide a robust way to evaluate how well IR object detection models perform under challenging, real-world conditions, simulating scenarios where sensor inputs might be degraded.
In addition to these benchmarks, a novel method called WiSE-OD (Weight-Space Ensembling for Object Detection) has been proposed. This innovative technique aims to leverage the complementary strengths of both RGB-trained and IR-trained models. WiSE-OD works by combining the “weights” (parameters) of an RGB zero-shot model (a model used directly without IR fine-tuning) and an IR fine-tuned model. This combination creates a new, more robust detector without requiring any additional training or increasing the computational cost during inference.
WiSE-OD comes in two main variants: WiSE-OD ZS, which blends the weights of an RGB zero-shot model and a fully fine-tuned IR model, and WiSE-OD LP, which combines a zero-shot model with one that has undergone linear probing (a form of fine-tuning where only the detection head is trained while the main feature extractor remains frozen). Extensive experiments were conducted using popular object detection architectures like Faster R-CNN, FCOS, and RetinaNet, comparing WiSE-OD against traditional fine-tuning strategies and other robust baselines.
The results demonstrate that WiSE-OD consistently improves both cross-modality robustness (adapting from RGB to IR) and corruption robustness (handling degraded images). For instance, on the LLVIP-C dataset, WiSE-OD ZS significantly boosted performance compared to traditional fine-tuning and linear probing. The method also showed remarkable stability in performance across different levels of corruption severity, unlike models that struggled significantly as corruption increased.
Qualitative analysis, using activation maps, further supported these findings, showing that WiSE-OD models were better at identifying and focusing on relevant object features even in heavily corrupted images. An ablation study on the mixing coefficient (λ), which determines the balance between the zero-shot and fine-tuned weights, revealed that a value of 0.5 generally provides a good trade-off, though specific corruptions might benefit from slight adjustments.
Also Read:
- MuStD Network: Improving Outdoor 3D Object Detection Through Multimodal Fusion
- Bridging the Divide: Enhancing Search Across Images and Text
While WiSE-OD offers a promising solution, it does have some limitations. It requires access to both a zero-shot and a fine-tuned model, and the fixed mixing coefficient might not be universally optimal. Future work could explore adaptive λ selection, ensembling more model variants, and testing against real-world IR degradations beyond synthetic corruptions. This research marks a significant step towards more reliable and robust object detection systems in challenging infrared environments. You can find the full research paper here.
