TLDR: A new, lightweight defense for Vision-Language Models (VLMs) uses tensor decomposition to filter adversarial noise from image representations without retraining. It significantly improves VLM robustness against attacks like PGD, with Tensor Train decomposition showing the best performance and efficiency on datasets like COCO and Flickr30K. This plug-and-play solution offers a practical way to enhance VLM security with minimal computational overhead.
Vision-Language Models (VLMs) have transformed how we interact with AI, enabling applications like image retrieval and visual question answering by understanding both images and text. Models such as CLIP have shown remarkable capabilities in learning joint embeddings of these different data types. However, these powerful models are not without their weaknesses; they are highly susceptible to adversarial attacks. These attacks involve making tiny, often imperceptible changes to an image that can drastically fool the model, leading to incorrect classifications or understandings.
Traditional methods to defend against these attacks, such as extensive adversarial training or significant architectural changes, often come with high costs. They might require massive computational resources for retraining on large datasets or demand substantial redevelopment efforts, making them impractical for many existing, large pre-trained VLMs.
A Novel, Lightweight Defense
Researchers have introduced a new, lightweight defense mechanism that tackles this vulnerability without the need for costly retraining or architectural modifications. This innovative approach leverages tensor decomposition, a mathematical technique for breaking down complex data structures, to filter out adversarial noise from the internal representations of VLMs. The core idea is that adversarial perturbations often introduce high-frequency noise into the model’s feature representations. By decomposing and then reconstructing these representations using low-rank tensor approximations, the defense can effectively filter out this noise while preserving the essential semantic content of the image.
The defense mechanism works by intercepting tensors at specific layers within the VLM’s vision encoder. It then applies one of three tensor decomposition methods: CP/PARAFAC, Tucker, or Tensor-Train Decomposition (TT). After decomposition, a residual connection is used to combine the original and reconstructed features, with a parameter (α) controlling the strength of this defense.
Also Read:
- Improving Text-to-Image Spatial Understanding Through Structured Information
- Cognitive-Inspired AI: A New Method for Attention Management in Transformers
Key Findings and Performance
The effectiveness of this tensor decomposition defense was rigorously tested using the CLIP model on two widely recognized datasets, MS-COCO and Flickr30K, against Projected Gradient Descent (PGD) adversarial attacks. The results were compelling:
-
Optimal Parameters: The defense performed best with lower residual strength values (α = 0.1 – 0.3), indicating that filtering more of the original features helps remove adversarial noise. Similarly, lower decomposition ranks (8-32) were found to be optimal, supporting the idea that adversarial noise resides in higher-frequency components that are discarded by low-rank approximations.
-
Superior Decomposition Method: Among the tested methods, Tensor Train (TT) decomposition consistently outperformed CP and Tucker. On Flickr30K, TT restored 12.3% of performance lost to attacks, raising Recall@1 accuracy from 7.5% to 19.8%. On MS-COCO, it recovered 8.1% performance, improving accuracy from 3.8% to 11.9%.
-
Targeting Specific Layers: Applying the defense to the final normalization layer of the vision encoder yielded the best results for single-layer configurations. Extending this to multi-layer configurations, specifically targeting the normalization layers of the last five transformer blocks, provided the strongest overall defense.
-
Computational Efficiency: The defense is also computationally efficient. Single-layer TT decomposition introduced only a 1.22x overhead, maintaining 82% throughput. Even a 5-layer TT configuration, offering robust defense, presented a reasonable 3.93x overhead, demonstrating a practical balance between security and performance.
This tensor decomposition defense offers a practical, plug-and-play solution for enhancing the robustness of existing Vision-Language Models. It provides significant improvements against adversarial attacks with minimal overhead and without the need for extensive retraining, making it a valuable tool for deploying more secure AI systems. For more details, you can refer to the full research paper here.
