TLDR: This research paper introduces a fundamental distinction in adversarial attacks: attacks in the input data-space can transfer between models, while attacks in the model’s internal representation space generally do not, unless the models’ internal representations are geometrically aligned. This hypothesis is supported by mathematical proofs and empirical evidence across image classifiers, language models, and vision-language models, explaining why some adversarial examples (like text jailbreaks) transfer easily, while others (like image jailbreaks in VLMs) struggle to spread.
The world of artificial intelligence is constantly evolving, and with its advancements come new challenges, particularly in the realm of security and robustness. A recent study delves into a puzzling aspect of adversarial attacks: why some attacks successfully transfer between different AI models, while others, seemingly similar, fail to do so. This research, titled UNDERSTANDING ADVERSARIAL TRANSFER: WHY REPRESENTATION-SPACE ATTACKS FAIL WHERE DATA-SPACE ATTACKS SUCCEED, proposes a fundamental distinction that sheds light on this phenomenon.
Authored by Isha Gupta, Rylan Schaeffer, Joshua Kazdan, Ken Ziyu Liu, and Sanmi Koyejo, the paper highlights that adversarial examples—inputs designed to trick AI models—have long been known to transfer between image classifiers and language models. For instance, a text-based ‘jailbreak’ designed for one language model might also work on another. However, recent observations showed that image-based jailbreaks against vision-language models (VLMs) did not exhibit this same transferability, leaving researchers puzzled.
The Core Hypothesis: Data-Space vs. Representation-Space Attacks
The central idea introduced by the researchers is that the success of an adversarial attack’s transferability depends on where the attack operates: either in the ‘input data-space’ or in the model’s ‘representation space’.
- Data-Space Attacks: These attacks involve perturbing the raw input data itself, like adding a subtle, imperceptible noise to an image or a specific suffix to a text prompt. Since different models often learn similar input-output relationships, even if their internal workings differ, these data-space perturbations tend to cause similar misbehaviors across models.
- Representation-Space Attacks: These attacks target the internal, abstract representations (or ‘features’) that a model learns from the input, rather than the raw input itself. Because different models can have vastly different internal architectures and ways of representing information, an attack optimized for one model’s internal representation is unlikely to work on another model unless their internal representation spaces are geometrically aligned.
Evidence Across AI Systems
The researchers provide both theoretical and empirical evidence to support their hypothesis across various AI settings:
- Mathematical Proof: In a simplified mathematical model, they demonstrated that if two networks compute the same input-output function but use different internal representations, data-space attacks transfer perfectly, while representation-space attacks require a strict geometric alignment to transfer.
- Image Classifiers: Experiments with ResNet18 image classifiers showed that universal adversarial perturbations applied to raw input images (data-space) successfully transferred to other classifiers. In contrast, attacks optimized on the internal latent representations of the models (representation-space) largely failed to transfer.
- Language Models (LMs): Textual jailbreaks, which modify the input text (data-space), are known to transfer between LMs. However, the study found that ‘soft prompt’ attacks, which are continuous embeddings directly injected into the model’s internal processing (representation-space), were overwhelmingly ineffective when transferred to different LMs.
- Vision-Language Models (VLMs): This is where the initial puzzle lay. The research confirmed that textual jailbreaks (data-space) successfully transferred between VLMs. Crucially, they showed that image jailbreaks, which act more like representation-space attacks from the VLM’s perspective, generally did not transfer.
When Representation-Space Attacks Can Transfer
The study also identified a critical condition under which representation-space attacks *can* transfer: when the models’ internal representations are geometrically aligned. This was demonstrated in two key scenarios:
- Finetuned Language Models: When soft prompt attacks were applied to different finetuned variants of the same base language model (e.g., Llama3-3B finetuned on different datasets), successful transfer was consistently observed. This is because these finetuned models, despite their differences, maintain highly similar latent representations.
- VLMs with Precise Layer Optimization: The researchers found that image jailbreaks could transfer between VLMs if the attacks were optimized at a very specific internal layer—the ‘post-projector’ latent representation, which is the point where the visual information is aligned with the language model’s input. This suggests that while the initial visual processing (the ‘projector’) might create unique representations, the subsequent language model components can have sufficiently aligned latent spaces to facilitate transfer.
Also Read:
- How Narrative Attacks Exploit Unified AI Models
- Challenging LLM Ownership: New Research Exposes Weaknesses in Fingerprinting Methods
Implications for AI Robustness
This research offers crucial insights for building more robust AI systems. It reveals that adversarial transfer is not a universal property of all attacks but is contingent on their operational domain. Understanding this distinction can help developers anticipate and defend against different types of attacks. For instance, defenses against data-space attacks might focus on input sanitization, while defenses against representation-space attacks might involve diversifying internal model architectures or actively misaligning latent spaces. The findings also contribute to a deeper understanding of how multimodal AI models process information, suggesting that modalities with ‘brittle, model-unique representations’ (like images in VLMs) might be more resistant to certain types of transfer attacks compared to more robustly consumed textual inputs.
