TLDR: A new research paper introduces an incremental training framework for Temporal Knowledge Graphs (TKGs) that improves predictions for rarely seen or new entities and combats model forgetting. The framework combines a model-agnostic enhancement layer, which uses global entity similarity to enrich representations, and a weighted sampling strategy that prioritizes training on less frequent entities. Experiments show significant improvements in link prediction performance and better handling of evolving knowledge graphs compared to existing methods.
Knowledge graphs, which structure information as interconnected entities and their relationships, have become vital for many AI applications, from question answering to powering large language models. Temporal Knowledge Graphs (TKGs) take this a step further by adding a time dimension, allowing them to capture how facts and relationships evolve over time. This is crucial for understanding dynamic real-world scenarios, like events unfolding in news streams or social media.
However, TKGs face significant challenges. They are often incomplete, with missing facts, and they are constantly evolving. This means models need to adapt to new information, including entities that have never been seen before or those with very few connections (known as ‘long-tail entities’). Traditional TKG models often struggle with these issues because they assume access to the entire graph during training and can suffer from ‘catastrophic forgetting,’ where learning new information causes them to forget previously acquired knowledge.
Researchers at USC Information Sciences Institute and Adobe Research have introduced an innovative incremental training framework designed to tackle these problems. Their approach focuses on improving predictions for long-tail and unseen entities in evolving TKGs, while also mitigating catastrophic forgetting. The core of their solution involves two main components: a model-agnostic enhancement layer and a weighted sampling strategy.
Enhancing Entity Representations with Global Similarity
One of the key innovations is the model-agnostic enhancement layer. Existing graph neural network (GNN) based methods typically rely on an entity’s immediate neighbors to create its representation. This works well for well-connected entities but falls short for sparse or new entities that lack rich local connections. The new enhancement layer addresses this by incorporating a broader, global definition of entity similarity.
Instead of just looking at direct connections, this layer identifies entities as similar if they participate in identical types of interactions, even if they aren’t directly linked. For example, if both ‘Da Vinci’ and ‘Van Gogh’ are subjects in ‘painted’ relations, the model recognizes their similarity based on this shared relational pattern. This allows the model to generalize knowledge to unseen or rare entities. The enhancement layer then combines the base entity representation from the underlying model with this globally enhanced information, giving more weight to entities with fewer connections.
Balancing Training with Weighted Sampling
The second crucial component is a weighted frequency-based sampling strategy. Standard training methods tend to overemphasize frequently occurring entities, leading to models that perform well on common data but poorly on rare entities. To counteract this, the new framework prioritizes training examples involving less common entities.
During training, quadruples (subject, relation, object, timestamp) involving rare entities are sampled more frequently. This is done by assigning a sampling weight inversely proportional to the frequency of the entities involved. To prevent overfitting to these rare patterns, a portion of the training examples are still selected uniformly. This dynamic sampling ensures that the model gets more opportunities to learn from long-tail patterns, adapting to the evolving entity distributions over time.
Also Read:
- Advancing Probabilistic Inference in Ordered Data
- Neuro-Symbolic AI Learns to Simplify Complex Formulas with Enhanced Efficiency
Promising Results and Future Directions
The researchers evaluated their framework on two benchmark datasets, ICEWS14 and ICEWS18, which record geopolitical interactions. They compared their method against existing approaches, including simple fine-tuning and other continual learning strategies like Experience Replay (ER) and Elastic Weight Consolidation (EWC).
The results were significant. The new framework consistently outperformed existing methods in overall link prediction, inductive link prediction (predicting links for unseen entities), and in addressing long-tail entities. Notably, the method achieved a 10% improvement in Mean Reciprocal Rank (MRR) on ICEWS14 and a 15% improvement on ICEWS18, which are standard metrics for link prediction. An ablation study confirmed that both the enhancement layer and the weighted sampling strategy contribute to these improvements, with the enhancement layer being particularly effective at mitigating catastrophic forgetting.
While this work marks a substantial step forward in TKG completion, especially for dynamic, real-world applications, the authors acknowledge that there’s still room for improvement, particularly for entities with extremely sparse neighborhoods. Future research could explore integrating richer features or continuously extracting facts from large language models to further refine TKG representations. For more details, you can refer to the full research paper.
