TLDR: This research introduces a unified framework using Activation Maximization (AM) to understand how deep neural networks (DNNs), including both CNNs and Vision Transformers (ViTs), learn and represent information. By synthesizing images that maximally activate specific neurons in both output and intermediate layers, the study reveals hierarchical feature learning and architectural differences. It also demonstrates how AM can generate adversarial examples, offering insights into model vulnerabilities.
Deep neural networks (DNNs) are at the heart of many advanced AI applications, from image recognition to natural language processing. But how do these complex systems truly “see” and understand the world? A new research paper titled “Representation Understanding via Activation Maximization” by Hongbo Zhu and Angelo Cangelosi from The University of Manchester offers a unified framework to shed light on this very question.
Inspired by neuroscience, where scientists probe biological neurons with stimuli to understand their function, this research uses a technique called Activation Maximization (AM). AM synthesizes artificial inputs that cause specific neurons within a DNN to respond most strongly. These optimized inputs, known as Activation Maximization Signals (AMS), reveal the underlying features that a neuron is most sensitive to. While previous efforts often focused only on the final output layer of Convolutional Neural Networks (CNNs), this work extends feature visualization to intermediate layers, providing a much deeper look into the hierarchical way DNNs build their understanding.
The paper proposes a unified framework that works for both traditional CNNs and the more recent Vision Transformers (ViTs). This is a significant step, as ViTs, with their attention-based mechanisms, have different internal workings compared to CNNs. Beyond just understanding representations, the researchers also explored how Activation Maximization can be used to create “adversarial examples”—subtly altered inputs that can trick a DNN into making incorrect classifications. This reveals potential weaknesses and decision boundaries within these powerful models.
How Activation Maximization Works
At its core, Activation Maximization treats the problem of finding the ideal input for a neuron as an optimization challenge. Imagine you want to find an image that makes a specific “cat” neuron in a network fire as much as possible. Instead of just looking through existing cat images, AM directly optimizes a blank input image by gradually adjusting its pixels based on the neuron’s “preferences.” This is done using a process similar to how neural networks learn, but instead of adjusting the network’s internal parameters, the input image itself is adjusted.
However, simply optimizing in the raw “pixel domain” often leads to noisy, unnatural-looking images that exploit the network’s sensitivity to high-frequency patterns. To overcome this, the researchers adopted an “optimization in the frequency domain” approach. By working with the image’s frequency components rather than individual pixels, the method naturally suppresses these high-frequency artifacts, resulting in more interpretable and natural-looking visualizations. This technique helps to generate “neural illusions” that are more aligned with what humans can perceive.
Exploring Different Layers and Architectures
The study applied its method to various CNN models like MobileNet, InceptionV3, and ResNet50V2, as well as the Vision Transformer ViT-B/16. They looked at two main areas: the “logits” layer (just before the final output, indicating class scores) and intermediate layers (which capture more abstract features).
Visualizations of the last logits layer revealed how different models represent semantic categories. For instance, all models could generate patterns for “Ladybug,” “Monarch Butterfly,” and “Pineapple” neurons. MobileNet tended to show repeated motifs, InceptionV3 produced more coherent and colorful patterns, and ResNet50V2 emphasized sharp, contrastive forms. ViT-B/16, in contrast, generated more abstract and diffuse visualizations, reflecting its global self-attention mechanism rather than localized filters. This showed that CNNs, with their spatial biases, tend to create more localized and interpretable patterns, while ViTs capture semantics in a more distributed way.
Delving into intermediate layers provided even deeper insights. For CNNs like MobileNet, early layers showed simple features like edges and textures, progressing to more complex geometric patterns and eventually to object-part-like textures in deeper layers. ResNet50V2, with its “skip connections,” demonstrated even greater global coherence and structural abstraction, building rich hierarchical representations. ViTs, however, showed a different progression. Even in early layers, they captured fine-grained textures and then quickly moved to structured, globally organized patterns, thanks to their attention mechanisms. This suggests that ViTs integrate long-range dependencies earlier in the network.
A key takeaway from comparing CNNs and ViTs is that CNNs build representations in a “bottom-up” fashion, while ViTs, through self-attention, create globally integrated features from the start. Interestingly, larger models like ViT-B/16 (with 86 million parameters) tended to produce more abstract feature representations compared to smaller models like MobileNet (4.2 million parameters), which can sometimes make them harder for humans to interpret.
Generating Adversarial Examples
The research also demonstrated the practical application of Activation Maximization in generating adversarial examples. By slightly perturbing an original image, the method could cause a model to misclassify it into a specific, incorrect target class (e.g., a ladybug image being classified as a monarch butterfly). These perturbations were designed to be imperceptible to humans. The study found that the perturbations generated for ViT models often showed a distinct grid-like pattern, possibly due to ViT’s patch-based input processing, which was less structured in CNNs. This highlights how AM can reveal vulnerabilities and the unique ways different architectures are susceptible to attacks.
Also Read:
- Unlocking Deeper Insights: How FMCE-Net++ Improves Neural Network Training
- Understanding AI’s Explanations: A Deep Dive into Attribution Theories
Looking Ahead
This research provides a powerful unified framework for understanding the internal representations of deep neural networks. While significant progress has been made, challenges remain, especially with the increasing size and complexity of modern AI models. Future work will focus on improving optimization strategies for large-scale ViTs, enhancing the human interpretability of the visualizations, and exploring how individual neurons interact to form complex representations. To learn more, you can read the full research paper here.
