TLDR: SAGE is a new framework for Continual Knowledge Graph Embedding (CKGE) that dynamically adjusts model capacity based on the evolving size of knowledge graphs. It uses scale estimation, lightweight embedding expansion, and dynamic distillation to balance integrating new information with preserving existing knowledge. Experiments show SAGE consistently outperforms other methods and effectively resists forgetting, offering a robust solution for dynamic knowledge graphs.
In the rapidly evolving digital landscape, knowledge graphs (KGs) are crucial for organizing vast amounts of information, powering applications from search engines to recommendation systems. Traditionally, these graphs were treated as static entities, meaning that if new information emerged, the entire system often had to be retrain from scratch. This process is not only time-consuming but also computationally expensive, especially for large, real-world KGs that are constantly growing with new entities, relationships, and facts.
To tackle this challenge, researchers have developed Continual Knowledge Graph Embedding (CKGE) methods. These methods aim to efficiently update KG embeddings – which are low-dimensional representations of entities and relations – to incorporate new facts while preserving the knowledge already learned. However, many existing CKGE approaches treat all updates uniformly, failing to account for the varying scales at which KGs evolve. Some updates might be minor, like adding a few new users, while others could be substantial, such as integrating entirely new marketplaces. This oversight can lead to suboptimal performance and a lack of systematic evaluation throughout the update process.
A new research paper introduces a novel framework called SAGE, which stands for Scale-Aware Gradual Evolution for Continual Knowledge Graph Embedding. SAGE is designed to dynamically adapt to the growth of knowledge graphs, ensuring that the model’s capacity aligns with the evolving data. The core idea behind SAGE is to adjust the embedding dimensions based on the scale of updates, rather than using a fixed size. This adaptive approach is inspired by observations in deep learning, where model performance often correlates with the number of parameters and the amount of data.
The SAGE framework operates in three main stages. First, it performs “Scale Estimation & Footprint Generation.” This involves determining the appropriate embedding dimensions based on the current scale of the knowledge graph. The researchers found a logarithmic relationship between the graph size and the optimal number of parameters, suggesting that as a KG grows, its embedding dimensions should also expand, but not necessarily linearly. This stage also generates “footprints” – statistical indicators that capture the importance and learning characteristics of each knowledge element, considering both novelty (how often an element appears in new facts) and historical reliance (its presence in existing facts and the quality of its previous representation).
The second stage is “Selective Replay & Embedding Expansion.” When the embedding dimensions need to be expanded, SAGE doesn’t just randomly add new dimensions. Instead, it uses a lightweight mechanism that leverages existing embeddings as input features for the expanded dimensions. This helps maintain semantic consistency and prevents information loss. To make this process efficient, SAGE focuses on “challenging samples” from the existing knowledge graph – those instances where the current representations show limitations. By replaying a small subset of these difficult examples, the model efficiently preserves learned knowledge while expanding its capacity.
Finally, the “Dynamic Distillation & Integration” stage optimizes the representations of all entities and relations, including both new and existing ones, based on the newly added triples. The novelty and reliance footprints generated in the first stage play a crucial role here, guiding the updates of existing weights. This dynamic weighting ensures a balanced optimization process, preventing the model from “catastrophically forgetting” old knowledge while effectively integrating new information. The framework uses a combined loss function that balances the learning of new knowledge with the preservation of historical knowledge.
Extensive experiments were conducted on seven different benchmark datasets, simulating various growth patterns of knowledge graphs. The results consistently show that SAGE outperforms existing baseline methods, demonstrating significant improvements in metrics like Mean Reciprocal Rank (MRR), Hits@1, and Hits@10. For instance, SAGE achieved a notable improvement of 1.38% in MRR, 1.25% in H@1, and 1.6% in H@10 on average across the datasets. The paper also highlights that SAGE achieves optimal performance at nearly every intermediate stage of the continual learning process, which is a significant advantage over methods that use fixed embedding dimensions.
The ablation studies further confirmed the importance of each component of SAGE, particularly the Lightweight Expansion and Dynamic Integration mechanisms, which were found to contribute most significantly to performance gains. The adaptive dimension strategy proved superior to static approaches, as fixed dimensions often either lacked capacity for growing graphs or led to underfitting in early stages with limited data. SAGE also demonstrated strong resistance to forgetting, maintaining high performance on previously learned knowledge while adapting to new facts.
Also Read:
- ReaGAN: Pioneering Autonomous Agentic Nodes for Advanced Graph Machine Learning
- Advancing GUI Automation with Difficulty-Aware Reinforcement Learning
In conclusion, SAGE offers an effective and scalable solution for Continual Knowledge Graph Embedding. By dynamically adapting its model capacity to the evolving scale of knowledge graphs and employing intelligent mechanisms for knowledge preservation and integration, SAGE provides a robust framework for handling the dynamic nature of real-world knowledge. This work not only advances the field of CKGE but also offers valuable insights for adaptive learning in other dynamic data environments. For more technical details, you can refer to the full research paper available here.
