TLDR: A new research paper reveals that catastrophic forgetting in deep continual learning is caused by an “implicit adversarial attack” where new-task training gradients align with sensitive directions of old-task knowledge. This “adversarial alignment” is driven by the low-rank bias of deep networks, which confines these directions to a common low-dimensional subspace. The authors propose ‘backGP,’ a novel method that extends existing Gradient Projection techniques to address alignment from backward propagation, significantly reducing forgetting and improving accuracy in continual learning.
Continual learning, a field striving to equip artificial intelligence with the human-like ability to acquire new skills sequentially without forgetting old ones, faces a significant hurdle known as catastrophic forgetting. This phenomenon occurs when training a model on a new task inadvertently overwrites the knowledge gained from previous tasks, leading to a severe decline in performance on those older tasks.
A recent research paper, titled “ON THE IMPLICIT ADVERSARIALITY OF CATASTROPHIC FORGETTING IN DEEP CONTINUAL LEARNING,” sheds new light on the fundamental cause of catastrophic forgetting in deep neural networks. Authored by Ze Peng, Jian Zhang, Jintao Guo, Lei Qi, Yang Gao, and Yinghuan Shi, the study reveals that the process of training a model on a new task is, surprisingly, an implicit adversarial attack against its existing knowledge.
The core finding is that the gradients generated during new-task training automatically and precisely align with the “sharp directions” of the old-task’s loss landscape. Imagine the loss landscape as a mountainous terrain where the goal is to find the lowest point (minimal loss). Sharp directions are like steep slopes where even a small step can lead to a drastic increase in loss. When new-task gradients align with these sharp directions, the old-task loss rapidly increases, causing catastrophic forgetting. This phenomenon is termed “adversarial alignment.”
What makes this discovery particularly intriguing is its counter-intuitive nature. The sharp directions in a high-dimensional weight space are incredibly sparse, making it statistically unlikely for new-task gradients to align with them purely by chance. The researchers explain this by introducing the concept of “low-rank bias.” They theoretically demonstrate that the training process itself induces a low-rank structure in the model’s weight matrices. These low-rank matrices, through both forward and backward propagation, effectively confine the new-task gradients and the old-task’s high-curvature directions into the same low-dimensional subspace. This confinement makes the adversarial alignment much easier to occur.
The depth of a neural network further intensifies this low-rankness and, consequently, the adversarial alignment. This explains why catastrophic forgetting behaves differently in deep networks compared to simpler, single-layer networks, where data properties play a more dominant role.
The paper also examines existing methods designed to mitigate forgetting, specifically Gradient Projection (GP) methods. While GP methods are effective at reducing adversarial alignment caused by forward propagation, they fall short in addressing the alignment that arises from backward propagation. To tackle this limitation, the researchers propose a novel strategy called “backGP.” This method extends GP techniques to the backward direction, further reducing the residual adversarial alignment.
Extensive experiments across various continual learning benchmarks and network architectures (including ResNet, VisionTransformer, and MLP-Mixer) confirm the existence and persistence of adversarial alignment. The results show that without this alignment, forgetting would be negligible. When combined with plasticity-enhancing techniques like spectral regularization, backGP significantly improves forgetting mitigation by 10.8% and boosts overall accuracy by 12.7% on average compared to traditional GP methods.
Beyond its direct impact on continual learning, this research has broader implications. It establishes a hidden connection between continual learning and adversarial robustness, suggesting that insights from adversarial attack research could be transferred to better understand and combat catastrophic forgetting. Furthermore, the analysis offers a preliminary model for how pretraining on one dataset can shape the learning of subsequent tasks, which is highly relevant for understanding the effectiveness of modern foundation models and parameter-efficient finetuning.
Also Read:
- AI’s Achilles’ Heel: How Adaptive Attacks Exploit Trusted Monitors
- Unpacking Titans: A Closer Look at a Test-Time Memory Model
For a deeper dive into the technical details and experimental setups, you can refer to the full research paper: On the Implicit Adversariality of Catastrophic Forgetting in Deep Continual Learning.
