TLDR: A new research paper introduces Scarf-DETR, a novel object detection model designed to overcome performance drops when infrared or visible data is incomplete or missing. It features a ‘Scarf Neck’ module with Modality-Agnostic Deformable Attention for flexible data processing and a ‘pseudo modality dropout’ training strategy for robustness. Scarf-DETR significantly improves detection accuracy in incomplete modality scenarios while maintaining high performance in complete modality settings, making it highly adaptable for real-world applications.
Infrared-Visible Object Detection (IVOD) is a critical technology for applications that need to operate around the clock, especially in challenging environments like low light or fog where standard visible cameras struggle. These systems combine information from both infrared and visible light cameras to get a more complete picture of the surroundings and detect objects.
However, a significant challenge for current IVOD models arises when one of these data streams is incomplete or entirely missing. This can happen due to sensor malfunctions, image blurriness, or overexposure in real-world scenarios. When a dominant modality is absent, the performance of these detectors can drop dramatically, limiting their practical use.
A new research paper, titled “On Modality Incomplete Infrared-Visible Object Detection: An Architecture Compatibility Perspective,” addresses this very problem. Authored by Shuo Yang, Yinghui Xing, Shizhou Zhang, and Zhilong Niu from Northwestern Polytechnical University, China, the paper introduces a novel solution called Scarf-DETR.
Introducing Scarf-DETR: A Flexible Solution
The core of their proposal is a ‘plug-and-play’ module called the Scarf Neck, designed specifically for DETR (Detection Transformer) variants, which are popular object detection models. This Scarf Neck module incorporates a clever mechanism called Modality-Agnostic Deformable Attention (MADA). MADA allows the detector to flexibly adapt to either single or double modalities during both training and inference. This means whether it receives both visible and infrared images, or just one of them, the system can still process the information effectively.
When both modalities are present, the Scarf Neck enhances features from each modality and intelligently combines them. In situations where one modality is missing, instead of simply duplicating the available data or failing, the system focuses on strengthening the features of the remaining modality, ensuring that valuable information is not lost.
Smart Training for Robustness
To further boost the detector’s ability to handle missing data, the researchers developed a ‘pseudo modality dropout’ strategy for training Scarf-DETR. Traditional dropout methods might discard entire images, reducing the diversity of training data. The pseudo modality dropout, however, selectively disconnects modality connections for some image pairs, effectively simulating missing modality scenarios without wasting any training samples. This strategy makes the detector compatible and robust across both single and double modality operating modes.
Comprehensive Evaluation and Impressive Results
To thoroughly test their approach, the team also created a comprehensive benchmark for modality-incomplete IVOD tasks. This benchmark assesses performance in various scenarios, including when the missing modality is either dominant or secondary. The results were compelling: Scarf-DETR not only performed exceptionally well in scenarios with missing modalities but also achieved superior performance on standard, complete-modality IVOD benchmarks.
For instance, on the LLVIP dataset, a previous state-of-the-art model saw its performance drop to just 15% mAP in a visible-only scenario. Scarf-DETR, however, achieved a remarkable 55.5% mAP in the same scenario, demonstrating a significant improvement in handling incomplete data. The method also showed more stable performance across various mixed-modality settings compared to other models.
Also Read:
- Unlocking Better Object Detection with External VFM Knowledge
- Enhancing Synthetic Infrared Images with Smart Inference Techniques
Looking Ahead
The Scarf Neck module’s flexible design means it can be easily integrated into existing DETR-series detectors, making it a valuable addition to the field. The researchers believe their modality-agnostic design has great potential for other multi-modal tasks, such as RGB-D (color and depth) or RGB-SAR (color and synthetic aperture radar) object detection, where similar challenges with missing data exist. For more technical details, you can refer to the full research paper here.
