TLDR: SEDEG is a novel two-stage framework for Vision Transformers that addresses catastrophic forgetting in incremental learning, especially in low-memory settings. The first stage enhances the decoder’s generality by training an ensembled encoder with feature boosting and balanced classification. The second stage improves the encoder’s generality by compressing the ensembled encoder using balanced and feature-based knowledge distillation. Extensive experiments show SEDEG’s superior performance in mitigating forgetting and adapting to new data.
The field of artificial intelligence is constantly evolving, and one significant challenge is enabling systems to learn new information without forgetting what they’ve already learned. This issue, known as “catastrophic forgetting,” is particularly problematic in incremental learning, where models are continuously updated with new data. Traditional deep neural networks often struggle with this, especially when memory is limited.
A new research paper introduces a novel solution called SEDEG (Sequential Enhancement of Decoder and Encoder’s Generality), a two-stage training framework designed for vision transformers (ViT). This framework aims to improve the generalization capabilities of both the encoder and decoder components of a neural network, which are crucial for retaining old knowledge while adapting to new data. Many existing methods tend to focus on only one of these components, limiting their effectiveness, especially in scenarios with small memory where only a limited number of past examples can be stored.
SEDEG tackles this limitation by sequentially enhancing the generality of both the decoder and the encoder. The framework is divided into two main stages.
Stage 1: Enhancing Decoder Generality
In the first stage, SEDEG focuses on learning an “ensembled encoder” and an “enhanced decoder.” This stage introduces an additional encoder that works alongside the existing one to capture features that the old encoder might have missed. These features are then combined and fed into the decoder. To further improve the decoder’s ability to generalize and to create a more balanced classifier, the researchers incorporated several key techniques. These include an auxiliary classification loss to help the supplementary encoder learn residual and boosted features, Task Embedding Distillation (TED) to prevent the decoder from forgetting previously learned task-level representations, and a balanced softmax classification loss to address class imbalance issues between new and old tasks.
Also Read:
- Unlocking Efficient Data Learning with Compressive Meta-Learning
- Enhancing Visual Perception with Decoupled Learning: Introducing DeCLIP
Stage 2: Enhancing Encoder Generality
The second stage of SEDEG is dedicated to “encoder enhancement.” Here, the goal is to compress the ensembled encoder back into a single, more generalized encoder while preserving the powerful feature extraction capabilities developed in the first stage. This is achieved through knowledge distillation strategies. Specifically, Feature Distillation (FD) is used to align the output features of the ensembled encoder with those of the new enhanced encoder, ensuring knowledge transfer. Additionally, a Balanced Logits Distillation (BLD) is employed to mitigate the class imbalance problem during distillation, ensuring the new model learns effectively from both new and old tasks. A notable aspect of this stage is that only the encoder is trained, keeping the decoder parameters frozen, which helps the model focus on feature extraction without losing stability.
Extensive experiments were conducted on three benchmark datasets: CIFAR100, Tiny-ImageNet200, and ImageNet100. The results consistently demonstrated SEDEG’s superior performance compared to state-of-the-art methods in class-incremental learning, particularly in small-memory scenarios. For instance, on the CIFAR100 dataset with 5 tasks, SEDEG showed a significant improvement in accuracy compared to previous methods. The paper also includes visualizations, such as t-SNE maps, which illustrate how SEDEG effectively distinguishes clusters of different categories, unlike other methods that show overlapping clusters.
Ablation studies confirmed the effectiveness of each component and trick introduced in both stages of SEDEG. The research also discusses the impact of class imbalance and memory size, highlighting that SEDEG’s strength lies in its ability to perform well even with low memory overhead, where class imbalance is more pronounced.
In conclusion, SEDEG offers a robust two-stage framework that effectively enhances the generalization capabilities of both the encoder and decoder in vision transformers for continual learning. This approach significantly mitigates catastrophic forgetting, especially in memory-constrained environments, paving the way for more adaptive and robust AI systems. For more in-depth information, you can read the full research paper available at this link.
