TLDR: GraphKeeper is a new framework for Graph Domain-Incremental Learning (Domain-IL) that addresses catastrophic forgetting when graph models learn from sequential, diverse domains. It tackles embedding shifts and decision boundary deviations through domain-specific parameter-efficient fine-tuning, intra- and inter-domain disentanglement, and deviation-free knowledge preservation using ridge regression. GraphKeeper achieves state-of-the-art performance, significantly outperforming existing methods and seamlessly integrating with Graph Foundation Models.
Graph incremental learning (GIL) is a rapidly evolving field focused on continuously updating graph models as new data becomes available. While traditional GIL methods have successfully tackled scenarios where new tasks or classes emerge within a single domain, a more complex challenge has arisen with the development of graph foundation models (GFMs): Graph Domain-Incremental Learning (Domain-IL).
Domain-IL involves updating models across multiple, distinct graph domains. This is crucial for GFMs, which need to integrate diverse graphs to build comprehensive and evolving knowledge bases. However, existing GIL methods, designed for single-domain scenarios, often struggle with Domain-IL, leading to a significant problem known as catastrophic forgetting. This means that when a model learns a new graph domain, it tends to forget the knowledge acquired from previous domains.
Understanding the Challenge: Catastrophic Forgetting in Domain-IL
The researchers behind the new paper, “GraphKeeper: Graph Domain-Incremental Learning via Knowledge Disentanglement and Preservation,” identified two primary causes for catastrophic forgetting in Domain-IL:
- Embedding Shifts: When a model adapts to new domains, its internal parameters change drastically. This can cause the learned representations (embeddings) of previously seen graphs to shift, leading to confusion between incremental graph domains.
- Decision Boundary Deviations: The way a model classifies data (its decision boundary) can also change significantly when learning new domains. This deviation can disrupt the model’s ability to accurately classify data from older domains.
Introducing GraphKeeper: A Novel Solution
To address these challenges, Zihao Guo and his team propose GraphKeeper, a novel GIL framework designed specifically for the Domain-IL scenario. GraphKeeper tackles catastrophic forgetting from two key perspectives: preventing embedding shifts and maintaining stable decision boundaries.
Multi-domain Graph Disentanglement
To prevent embedding shifts and confusion across different graph domains, GraphKeeper introduces a multi-domain graph disentanglement mechanism:
- Domain-Specific Parameter-Efficient Fine-Tuning (PEFT): Inspired by techniques like LoRA, GraphKeeper equips the pre-trained graph neural network (GNN) with domain-specific PEFT modules. When a new domain is learned, the parameters for previous domains are frozen. This ensures that the embeddings of previously learned domains remain stable and unaffected by new learning.
- Intra-domain Disentanglement: Within a single domain, GraphKeeper uses a contrastive learning objective to make node embeddings highly discriminable across different classes. This ensures that the model can clearly distinguish between different types of nodes within the same domain.
- Inter-domain Disentanglement: To prevent overlap and confusion between different domains, GraphKeeper pushes the embeddings of the current domain away from the representative prototypes of all previously learned domains. This ensures clear semantic separation in the embedding space.
Deviation-Free Knowledge Preservation
To maintain a stable decision boundary, GraphKeeper separates the classification module from the embedding model. It uses a ridge regression-based knowledge preservation mechanism that does not rely on gradient updates through back-propagation. This approach allows the model to continuously fit incremental domains while keeping the decision boundary stable, thus avoiding catastrophic forgetting caused by its deviation. Crucially, this is achieved through a recursive update process that doesn’t require access to historical data, which is a common limitation in incremental learning.
Domain-Aware Distribution Discrimination
For graphs whose domains are not immediately observable (e.g., during testing), GraphKeeper employs a domain-aware distribution discrimination mechanism. It transforms features into a high-dimensional space using a randomly initialized GNN, which helps to separate domain prototypes. The test graph is then matched to the nearest domain prototype, ensuring that the correct domain-specific PEFT module is used for precise embedding and prediction.
Also Read:
- GraphChain: A New Approach for Large Language Models to Analyze Complex Graph Data
- A New Approach to Fine-Tuning LLMs: Random Initialization of Gated Sparse Adapters
Experimental Validation and Impact
Extensive experiments on 15 real-world datasets demonstrate that GraphKeeper achieves state-of-the-art results, showing a significant improvement of 6.5% to 16.6% over the next best method, with negligible forgetting. The research also highlights GraphKeeper’s broad applicability by showing it can be seamlessly integrated with existing Graph Foundation Models (GFMs) like GCOPE and MDGPT, enhancing their continuous updating capabilities without requiring memory replay, which can be a bottleneck for large graph corpora. An ablation study further confirmed the critical role of each component in GraphKeeper’s robust performance.
This work marks a significant step forward in graph incremental learning, particularly for the challenging Domain-IL scenario. By effectively mitigating embedding shifts and decision boundary deviations, GraphKeeper paves the way for more powerful and continuously evolving graph models. You can read the full research paper here: GraphKeeper: Graph Domain-Incremental Learning via Knowledge Disentanglement and Preservation.
