TLDR: CBPNet is a novel framework for continual learning on edge devices that tackles “plasticity loss,” a problem where models lose their ability to learn new information over time, especially in prompt-based methods with frozen backbones. It introduces an Efficient CBP Block that adaptively re-initializes underutilized parameters, thereby restoring the model’s learning vitality. This approach significantly improves average accuracy on benchmarks like Split CIFAR-100 and Split ImageNet-R, achieving state-of-the-art results with minimal additional parameters (less than 0.2% of the backbone’s size), making it highly efficient and practical for resource-constrained edge devices.
In the rapidly evolving world of artificial intelligence, applications like robotics and autonomous driving demand models that can learn continuously and adapt to new information in real-time, especially on resource-constrained edge devices. This field, known as continual learning, faces a significant hurdle: catastrophic forgetting, where models tend to lose previously acquired knowledge when learning new tasks.
While pre-trained models combined with prompt-based methods have emerged as a popular strategy to combat catastrophic forgetting, they introduce a new challenge: plasticity loss. This refers to the model’s diminishing ability to learn new information over time, primarily due to the core model being frozen and the limited capacity of the prompt parameters. Essentially, the model loses its ‘learning vitality’.
Introducing CBPNet: A Solution for Plasticity Loss
A new research paper, “CBPNET: A CONTINUAL BACKPROPAGATION PROMPT NETWORK FOR ALLEVIATING PLASTICITY LOSS ON EDGE DEVICES”, proposes an innovative solution called the Continual Backpropagation Prompt Network (CBPNet). Developed by Runjie Shao, Boyu Diao, Zijia An, Ruiqi Liu, and Yongjun Xu from the Institute of Computing Technology, Chinese Academy of Sciences, and the University of Chinese Academy of Sciences, CBPNet is designed to restore a model’s learning vitality efficiently and with minimal additional parameters.
The core idea behind CBPNet is that plasticity loss stems from a lack of update activity in underutilized parameters during the training process. To counteract this, CBPNet integrates an “Efficient CBP Block” into a prompt-based framework (specifically, DualPrompt). This block adaptively monitors and re-initializes neurons that have minimal contribution or are ‘stagnant’, effectively injecting new life into the network without compromising existing knowledge.
How CBPNet Works
CBPNet builds upon the DualPrompt framework, which uses lightweight, learnable prompts to guide a frozen pre-trained Vision Transformer (ViT) model. It employs both task-shared G-Prompts for common knowledge and task-specific E-Prompts for new information. The Efficient CBP Block is strategically placed after the ViT feature extractor and before the final classification head. It’s a modular, post-hoc block, meaning it doesn’t alter the main pre-trained model, ensuring safety and efficiency.
The block quantifies the “contribution utility” of its internal units (neurons) to identify those that are no longer contributing meaningfully. Units that are mature (exceeding a certain age threshold) and have low utility are then selectively re-initialized. This re-initialization involves resampling input weights and zeroing out output weights, allowing these units to learn new patterns without disrupting prior knowledge.
Also Read:
- Lightweight Image Captioning with Enhanced Relational Understanding
- Tensor Decomposition: A Lightweight Shield for Vision-Language Models Against Adversarial Attacks
Efficiency and Performance
CBPNet is designed with efficiency in mind, making it highly suitable for edge devices. It’s a rehearsal-free method, meaning it doesn’t require storing past data, which is a major advantage for memory-constrained environments. The additional parameters introduced by CBPNet (prompts and the CBP block) constitute less than 0.2% of the frozen backbone’s size, keeping the model lightweight.
Experimental results demonstrate CBPNet’s effectiveness across multiple benchmarks. On Split CIFAR-100, it improved average accuracy by over 1% compared to a strong baseline. On the more challenging Split ImageNet-R, it achieved a state-of-the-art accuracy of 69.41%. Crucially, CBPNet’s accuracy curve remains significantly more stable in later learning stages compared to other methods, indicating its success in mitigating plasticity loss as tasks accumulate.
In conclusion, CBPNet offers a simple yet effective solution for deploying high-performance and adaptive AI systems on resource-constrained edge devices. By intelligently restoring the learning vitality of models, it paves the way for more robust and continually learning AI applications in dynamic environments.
