TLDR: Fed-LSCL is a novel framework for Federated Continual Learning that addresses the challenges of deploying large Foundation Models (FMs) in resource-constrained and privacy-sensitive environments. It proposes a collaboration between lightweight local models and a large, frozen FM. The small models continually adapt to new tasks and generate parameters to fine-tune the FM, while a server-side distillation process fuses knowledge from diverse clients. This approach effectively mitigates catastrophic forgetting, enhances privacy, and significantly reduces communication and computational overhead, demonstrating superior performance on various datasets.
Foundation Models (FMs), often referred to as large models like GPT-4 and LLaMa, have revolutionized artificial intelligence with their impressive ability to handle diverse tasks in areas such as natural language processing and computer vision. Their powerful generalization capabilities make them appealing for Federated Learning (FL), a method where AI models are trained across multiple decentralized devices or servers holding local data samples, without exchanging the data itself.
However, deploying these massive FMs in real-world scenarios, especially in a dynamic setting known as Federated Continual Learning (FCL), presents significant hurdles. FCL allows each participant to continuously learn from a private stream of evolving tasks without forgetting previously acquired knowledge. The challenges arise because client data is private and cannot be directly shared with FMs. Furthermore, clients often have limited computational resources, making it impractical to train or even adapt entire large models locally. This leads to suboptimal performance on specific local tasks. A major problem is also ‘catastrophic forgetting,’ where models tend to forget old knowledge when learning new tasks, a challenge amplified by the immense size and complexity of FMs.
To bridge this gap, researchers have introduced Fed-LSCL, a pioneering collaborative framework that integrates powerful, yet heavy, FMs with lightweight and flexible local ‘Domain Models’ (DMs), which are essentially small models. This innovative approach allows the small local models to act as a dynamic bridge, continually adapting to new tasks while simultaneously enhancing the utility of the large foundation model. The small models, being trained locally on private data, can better leverage sensitive information, and their smaller size makes them compatible with more mature continual learning techniques.
How Fed-LSCL Works: A Three-Part Harmony
The Fed-LSCL framework is built upon three novel components:
First, Large-Small Model Collaborative Training transforms the complex task of continually fine-tuning a large model into a more manageable process. Instead of directly updating the large FM, the small local model learns to generate ‘sample-aware adapter parameters’ for the fixed FM. These parameters are then injected into the FM to adjust its representations, effectively allowing the small model to guide the large model’s learning without the large model itself undergoing direct, resource-intensive training. This significantly reduces the computational and communication burden on client devices.
Second, Small Model Continual Fine-tuning focuses on preventing the small local models from forgetting past knowledge. It achieves this by saving the previous version of the local small model and its parameter generator. When learning new tasks, the current small model is trained to generate consistent features and parameters for past data samples stored in a local buffer. This mechanism ensures that the small model retains its ability to fine-tune the large model for previously learned tasks, mitigating temporal forgetting.
Finally, One-by-One Distillation (O2D) handles the aggregation of knowledge on the server side. Unlike traditional federated learning that might create a single global model, O2D generates a unique, personalized global model for each client. Each client’s parameter generator acts as a ‘student’ model, learning from the parameter generators of all other clients, which serve as ‘teacher’ models. This process distills collective knowledge into each personalized model, preventing performance degradation on local tasks while enhancing generalization. Crucially, only the local buffer feature set and the parameter generator are uploaded, not the entire local CNN, significantly enhancing privacy and reducing communication overhead.
Also Read:
- Oblivionis: Enabling Data Forgetting in Collaborative AI Models
- A New Toolkit for Multimodal AI: Introducing MCITlib for Continuous Learning
Performance and Practical Benefits
Extensive experiments demonstrate that Fed-LSCL achieves state-of-the-art performance on challenging datasets like CIFAR-100 and ImageNet-R, showing significant improvements, especially on the latter. Its performance remains robust even as the number of tasks increases, highlighting its effectiveness in mitigating catastrophic forgetting. The framework also proves resilient to heterogeneity in local models, meaning clients can use different small model architectures (e.g., ResNet-18, MobileNetV3, VGG-16) and still achieve strong results.
From a practical standpoint, Fed-LSCL offers strong privacy protection by keeping the locally trained CNNs on client devices and only uploading distilled feature sets and parameter generators. This makes it much harder to infer private data. Furthermore, the communication and computational costs are significantly reduced, as only lightweight components are exchanged and processed. This makes Fed-LSCL particularly well-suited for resource-constrained environments like Edge-AI devices.
In essence, Fed-LSCL represents a significant step forward in Federated Continual Learning, offering a robust, efficient, and privacy-preserving solution for enabling large foundation models to continually adapt and learn in decentralized, dynamic environments. You can read the full research paper here.
