TLDR: A study investigated the robustness of CLIP models for military vehicle classification under various occlusions. It found that dispersed occlusions (rain, snow) degrade performance more than contiguous ones, Transformer-based models outperform CNNs, improved pre-training enhances robustness, and fine-tuning improves clean-image accuracy but can reduce occlusion robustness, especially with linear probing. The research emphasizes the need for occlusion-specific training strategies for real-world defense applications.
Vision-language models (VLMs) like CLIP have shown great promise in various applications, especially in scenarios where extensive labeled data is scarce. These models work by aligning images and text in a shared embedding space, allowing for zero-shot classification – meaning they can classify new objects without prior training examples. This capability is particularly appealing for defense applications, such as military vehicle classification, where new vehicle variants appear regularly and obtaining vast, labeled datasets for every condition is challenging.
However, a critical question arises: how robust are these models when faced with the challenging conditions of real-world military operational environments? CLIP, for instance, is primarily trained on high-quality internet imagery, which often doesn’t reflect the degraded visual conditions found in military settings. These conditions include partial occlusion from smoke, dust, foliage, structures, or adverse weather like rain and snow, all of which can significantly obscure vehicle features.
A recent research paper, “Occlusion Robustness of CLIP for Military Vehicle Classification”, by Jan Erik van Woerden, Gertjan Burghouts, Lotte Nijskens, Alma M. Liezenga, Sabina van Rooij, Frank Ruis, and Hugo J. Kuijf, delves into this underexplored area. The researchers investigated the robustness of several CLIP variants to various types of occlusion using a custom dataset of 18 military vehicle classes. They simulated both contiguous occlusions (like a vehicle emerging from behind a tree, modeled by a sliding blackout or bar occlusion) and dispersed occlusions (such as rain, snow, or grid dropout, mimicking environmental interference or sensor issues).
To evaluate robustness, they used Normalized Area Under the Curve (NAUC) across different occlusion percentages, alongside standard accuracy. The NAUC metric is particularly useful as it quantifies performance degradation across the entire spectrum of occlusion levels, providing a single scalar value for comparison.
Key Findings from the Research
The study yielded four significant insights into CLIP’s performance under occlusion:
1. Dispersed vs. Contiguous Occlusions: Fine-grained, dispersed occlusions (like snow and rain) degrade model performance more significantly than larger, contiguous occlusions. For example, a PE-Core-ViT-L/14-336 model achieved an NAUC of 61.3% for dispersed occlusions compared to 78.9% for contiguous occlusions. This suggests that when occlusions are spread out and affect many small parts of an image, models struggle more than when a large, connected area is obscured, allowing the model to focus on the remaining visible parts.
2. Architectural Differences: Transformer-based CLIP models consistently outperformed CNN-based CLIP models. Vision Transformers (ViT) showed a clear advantage, with ViT-B/16 achieving up to 22 percentage points higher NAUC than ResNet50. This is attributed to the global attention mechanism of Transformers, which allows them to dynamically balance relevant features across the entire image, even when parts are occluded, unlike CNNs that rely more on local receptive fields.
3. Impact of Pre-training Methodology: The way a model is pre-trained significantly affects its robustness. Models from the Perception Encoder-Core (PE-Core) methodology consistently outperformed CLIPA counterparts at similar scales. For instance, PE-Core models showed a +6.7 percentage point NAUC improvement at around 320 million parameters. This indicates that improved pre-training, often involving more diverse augmentations and better datasets, can enhance robustness beyond just scaling up the model size.
4. Fine-tuning Trade-offs: Fine-tuning, while boosting accuracy on clean images, introduced a trade-off with robustness. Linear probing, which only trains the classification head while freezing the visual encoder, significantly improved clean-image accuracy (from 55.6% to 88.0%) but drastically reduced robustness under dispersed occlusions (snow NAUC dropped from 54.0% to 36.0%). Full fine-tuning, which updates all model parameters, mitigated this effect (snow NAUC of 44.5%) but still fell short of the zero-shot model’s consistency in robustness. This suggests that robustness is built in the earlier layers of the visual backbone, and simply adapting the high-level features might not recover crucial spatial cues lost to occlusion.
Also Read:
- Bridging Vision and Language: The Evolution of Object Detection with Advanced AI Models
- Unpacking Compositional Understanding in Vision-Language Models
Implications for Real-World Deployment
These findings underscore that for reliable deployment in defense applications, occlusion robustness cannot be assumed from baseline accuracy or model size alone. The research highlights the need for occlusion-specific augmentations during training, such as grid masks, rain streaks, or sliding blocks, to mimic real-world occlusion patterns. Furthermore, masking-based training strategies and occlusion-aware loss objectives could encourage global context awareness and reduce sensitivity to corrupted regions, making vision-language models like CLIP more resilient in challenging military environments where partial visibility is common.
