TLDR: This research paper introduces Multi-Level Knowledge Distillation (MLKD) and Dynamic Self-Supervised Learning (SSL) to improve continual learning, specifically in Class-Incremental Learning with Repetition (CIR) scenarios. MLKD enhances model stability by distilling knowledge from multiple previous models across various perspectives, while Dynamic SSL boosts plasticity by utilizing unlabeled data for faster learning of new classes, with dynamic weighting to maintain focus on the primary task. The combined approach achieved 2nd place in the CVPR 2024 5th CLVISION Challenge, demonstrating significant performance improvements by effectively leveraging external unlabeled data.
In the evolving landscape of artificial intelligence, models often face the challenge of learning new information without forgetting what they’ve already learned. This is particularly true in ‘continual learning’ scenarios. A recent technical report, presented for the CVPR 2024 5th CLVISION Challenge, introduces innovative approaches to tackle this problem, specifically focusing on a more realistic setup called Class-Incremental Learning with Repetition (CIR).
Traditionally, continual learning assumes that new tasks introduce entirely new classes. However, CIR acknowledges that in real-world situations, models might encounter a mix of both new and previously seen classes. This setup also assumes easy access to abundant unlabeled data, for instance, from the internet. The core challenge here is to ensure models remain stable, retaining old knowledge, while also being plastic enough to quickly adapt to new information.
Many existing methods, known as rehearsal-based approaches, store a small subset of old data and replay it during training. While effective, these methods can be impractical due to privacy concerns or storage limitations. This paper explores a promising alternative: leveraging open-source external data that can be used during training and then discarded, bypassing storage and privacy issues.
Introducing Multi-Level Knowledge Distillation (MLKD)
To enhance the stability of models in CIR, the researchers propose Multi-Level Knowledge Distillation (MLKD). This technique goes beyond standard knowledge distillation by extracting knowledge from multiple previous versions of the model, considering various aspects like features and logits (the raw outputs before classification). For feature distillation, an L2 loss is applied to ensure consistent representations over time. For logit distillation, a more sophisticated mechanism using Gram matrices is introduced. This helps capture correlations within batches and across classes, which is crucial because unlabeled data might not perfectly reflect the distribution of previously trained data. To enrich the knowledge transfer, multiple previous models are maintained, and their outputs are mixed for distillation. These previous models are also gradually updated using exponential moving average (EMA) with the current model, promoting adaptability.
Dynamic Self-Supervised Learning (SSL)
To boost the model’s plasticity and accelerate the learning of new classes, the paper incorporates dynamic self-supervised learning (SSL) using the readily available unlabeled data. SSL encourages the model to learn general features that are useful across different tasks, which is vital for adapting to new concepts without disrupting existing knowledge. However, simply applying SSL can sometimes detract from the primary task, such as image classification. To address this, a dynamic weighting mechanism is applied to the SSL loss. This ensures that the model remains focused on its main objective while still benefiting from the generalization capabilities of SSL. The weight of the SSL decreases over tasks, allowing the model to prioritize the primary task as it progresses.
Also Read:
- Advancing Incremental Learning: A Two-Stage Approach to Enhance AI Model Generality
- Advancing Medical AI: A New Approach to Continual Learning with Efficient Prompt Management
Experimental Validation and Results
The proposed methods were tested in the CVPR 2024 5th CLVISION Challenge, which involved a continual learning setup with 50 experiences. The dataset was a subset of ImageNet-1K, comprising 130 classes. For each experience, 500 labeled and 1000 unlabeled data samples were provided. The model architecture used was ResNet-18, with specific constraints on GPU memory and training time. The researchers implemented their method using PyTorch and Avalanche libraries.
The results were compelling. The combination of MLKD and dynamic SSL achieved a final accuracy of 42.00%, securing 2nd place in the challenge. This was a significant improvement over a naive fine-tuning method (5.82%) and a strong baseline (19.54%). Adding dynamic SSL alone improved the baseline to 23.82%, while incorporating MLKD alone boosted it to 39.82%, highlighting the individual effectiveness of each component. Further ablation studies confirmed the synergistic effect of MLKD’s components, including feature KD, EMA, correlation-based logit KD, and using multiple previous models. The optimal number of previous models was found to be three, balancing knowledge retention with training efficiency. The dynamic weighting of SSL also proved crucial, improving accuracy from 40.76% to 42.00% compared to a static weighting approach. The paper also explored different self-supervised loss methods, finding that simple rotation prediction performed best under the challenge’s constraints.
In conclusion, this research presents a robust and effective framework for continual learning in realistic scenarios where unlabeled data is abundant. By combining multi-level knowledge distillation for stability and dynamic self-supervised learning for plasticity, the approach efficiently utilizes external unlabeled data to maintain previous knowledge and accelerate the learning of new classes, offering a promising direction for future continual learning systems. You can find more details about this work at the research paper link.
