TLDR: Neural networks often pack more features than they have dimensions (superposition), leading to interference between these features. This research shows that adversarial attacks exploit this interference, making attack patterns predictable and explaining why attacks transfer between models. The study demonstrates this in both synthetic and realistic vision models, suggesting that adversarial vulnerability can be an inherent consequence of a network’s efficient data compression, rather than just learning flaws or non-robust input features. It also notes that while superposition is a sufficient cause, other vulnerabilities like ‘algorithmic brittleness’ can exist even without superposition.
Adversarial examples, those subtly altered inputs that can trick even the most advanced neural networks, have long puzzled researchers. Why do these tiny, often imperceptible changes cause such drastic shifts in a model’s predictions? A new research paper, titled “ADVERSARIAL ATTACKS LEVERAGE INTERFERENCE BETWEEN FEATURES IN SUPERPOSITION,” offers a compelling new perspective, suggesting that these vulnerabilities might not be bugs, but rather a byproduct of how neural networks efficiently store and process information.
The paper, authored by Edward Stevinson, Lucas Prieto, Melih Barsbey, and Tolga Birdal from Imperial College London, argues that adversarial vulnerability can arise from a phenomenon called ‘superposition.’ Imagine a neural network trying to represent many different concepts or ‘features’ – like the presence of a cat’s ears, a dog’s tail, or a car’s wheel – but having fewer internal dimensions (like neurons) than the number of features it needs to represent. To handle this, the network ‘packs’ multiple features into shared dimensions, a process known as superposition.
While superposition is an incredibly efficient way for networks to store a vast amount of information, it comes at a cost: interference. When features are packed together, activating one feature can inadvertently affect others that share the same internal space. The researchers demonstrate that adversarial attacks specifically exploit this interference. Instead of being random, the patterns of these attacks are predictable, directly stemming from how these superposed features are arranged within the network.
Unpacking the Mechanism
The core idea draws on the ‘Linear Representation Hypothesis,’ which posits that neural networks represent fundamental input features as linear directions in their internal activation space. When networks use superposition, these feature directions become non-orthogonal, meaning they are not perfectly independent. This non-orthogonality is the root of the interference that adversaries can leverage.
The team conducted experiments using synthetic models where they could precisely control the degree of superposition. They found that adversarial perturbations didn’t just randomly alter inputs; they systematically exploited the geometric relationships between superposed features. The sign and magnitude of an input perturbation were strongly correlated with how a feature’s latent representation aligned with the ‘latent attack vector’ – the direction in the network’s internal space that leads to misclassification.
A crucial finding was how input correlations influence this internal geometry. When training data had strong correlations between features, the networks consistently learned similar geometric arrangements of superposed features across different training runs. This consistency, in turn, directly explained why adversarial attacks often ‘transfer’ successfully from one model to another, even if they were trained independently. If two models have similar internal feature arrangements, an attack designed for one will likely work on the other because it exploits the same interference patterns.
Conversely, the researchers showed that reducing superposition – for instance, by giving the network enough dimensions to represent each feature orthogonally – significantly limited adversarial vulnerability. In such cases, perturbations that changed the model’s prediction also genuinely changed the ground truth of the input, meaning true adversarial examples (where the ground truth remains the same) became virtually impossible to create.
Also Read:
- Unmasking the Adversarial Nature of Catastrophic Forgetting in Deep Neural Networks
- Mimicking Human Vision to Fortify AI Against Adversarial Attacks
Beyond Toy Models: Real-World Implications
To validate their findings, the researchers extended their analysis to a more realistic Vision Transformer (ViT) model trained on the CIFAR-10 image dataset. By introducing an engineered bottleneck layer, they forced the network’s class representations into superposition. The results mirrored those from the synthetic models: as the degree of superposition increased (i.e., the bottleneck dimension decreased), the model’s robust accuracy declined, and attack transferability between different model initializations increased. This suggests that the principles observed in simplified settings hold true for more complex, practical neural networks.
Interestingly, the paper also explores a scenario where adversarial vulnerability arises even without superposition, a concept they term ‘algorithmic brittleness.’ In a modular arithmetic task, the network learned a trigonometric algorithm with orthogonally represented features. However, attacks could still succeed by perturbing specific ‘key frequencies’ that the algorithm relied on. This highlights that while superposition is a sufficient cause for vulnerability, it is not the only one, and a deeper understanding of a network’s internal computations can reveal other attack vectors.
This research offers a mechanistic explanation for adversarial vulnerability, framing it as an inherent consequence of a network’s representational compression rather than solely a flaw in the learning process or non-robust input features. By understanding how interference patterns dictate attack characteristics, this framework could pave the way for designing more principled and semantically informed defenses against adversarial attacks. You can read the full research paper here.
