TLDR: The Relational Graph Perceiver (RGP) is a new graph transformer architecture designed for relational deep learning. It addresses limitations of existing models by introducing a temporal subgraph sampler to capture context from temporally proximate but structurally distant entities. RGP uses a Perceiver-style cross-attention bottleneck for efficient integration of structural and temporal information, making it scalable. Additionally, it features a flexible multi-task decoder that enables joint learning across diverse tasks with text-encoded labels, eliminating the need for task-specific output heads. Experiments show RGP achieves state-of-the-art performance on various benchmarks while offering significant computational efficiency.
Relational data, which captures complex interactions between entities over time, is fundamental to many real-world systems across industries like healthcare, finance, and e-commerce. Think of patient-provider interactions, customer purchases, or supply chain movements – these all generate vast amounts of interconnected information that evolves dynamically. Effectively modeling this data is crucial for making accurate predictions and informed decisions.
Traditional approaches to analyzing relational data, often converting it into heterogeneous temporal graphs, have faced significant challenges. Graph Neural Networks (GNNs), while good at capturing local structures, struggle with long-range dependencies and can suffer from issues like ‘oversquashing’ where information gets compressed too much. More recent Graph Transformers (GTs) offer a promising alternative by using attention mechanisms to reason across distant parts of a graph. However, both GNNs and existing GTs primarily focus on the structural relationships within the data, often treating temporal information merely as a constraint to filter out future events, rather than a rich signal for modeling. Furthermore, these models are typically designed for single predictive tasks, requiring separate models for each objective.
Introducing the Relational Graph Perceiver (RGP)
To address these limitations, a new architecture called the Relational Graph Perceiver (RGP) has been introduced. This innovative graph transformer aims to integrate both long-range structural and temporal dependencies more effectively, while also supporting multiple predictive tasks within a single model. The RGP framework is built around three core components: a novel temporal subgraph sampler, a Perceiver-style encoder, and a flexible multi-task decoder.
Capturing Temporal Context Beyond Direct Connections
One of RGP’s key innovations is its temporal subgraph sampler. In many real-world scenarios, events that are close in time might occur between entities that are not directly connected in the graph. For example, a major market shift could impact multiple customer accounts without any direct transactional links between them. Traditional sampling methods, which focus on immediate structural neighborhoods and use time only to prevent ‘looking into the future,’ miss these crucial, temporally correlated but structurally distant events.
The RGP’s temporal sampler explicitly leverages temporal proximity as a signal, independent of graph connectivity. It selects edges and their associated nodes based on their closeness in time to a reference point, allowing the model to incorporate broader contextual information like concurrent user activity or market trends. This complements standard structural sampling, providing a richer, more comprehensive view of the relevant context for prediction.
Efficiently Integrating Diverse Information
To handle this expanded context without incurring the prohibitive computational cost of full self-attention (which scales quadratically with the number of nodes), RGP employs a Perceiver-inspired encoder. This encoder uses a cross-attention-based latent bottleneck. Essentially, it compresses both the structural and temporal node embeddings into a fixed, smaller set of ‘latent’ representations. These latent tokens then attend to all nodes in the sampled subgraph, efficiently integrating information from the entire graph without requiring every node to interact directly with every other node. This design significantly reduces computational overhead, making RGP scalable for large relational datasets.
The encoder features two parallel cross-attention branches – one for temporally sampled nodes and another for structurally sampled nodes. The resulting latent representations from these branches are then combined, allowing the model to seamlessly integrate information from both local structural neighborhoods and broader, temporally correlated regions of the graph.
A Unified Approach to Multi-Task Learning
Another significant contribution of RGP is its flexible multi-task decoder. Unlike many existing models that require separate output layers or even entirely separate models for each predictive task, RGP uses a shared decoding mechanism. This decoder conditions predictions on both a task description (encoded from text) and the target node’s representation. It then compares this task-conditioned representation to text-encoded labels using a similarity-based objective. This unified framework allows a single RGP model to learn and perform multiple tasks simultaneously, even if those tasks have different label spaces, leading to greater efficiency and parameter sharing.
Also Read:
- GMoPE: Enhancing Graph Foundation Models with Specialized Experts and Efficient Prompting
- Unlocking AI’s Combinatorial Reasoning: A New Evaluation and Model Architecture for Compositional Generalization
State-of-the-Art Performance and Efficiency
Experiments conducted on diverse benchmarks like RelBench (binary classification), CTU (multi-class classification), and SALT (ranking-based tasks) demonstrate RGP’s superior performance. It consistently achieves state-of-the-art results, outperforming strong baselines like RelGT, DBFormer, and HGT across various domains. For instance, RGP showed an average relative improvement of 2.2% over RelGT on RelBench tasks. Crucially, RGP achieves these gains with significantly improved computational efficiency, scaling near-linearly with the number of input nodes, making it 2-6 times more compute-efficient than full self-attention models.
Ablation studies confirmed the importance of each component: removing the temporal sampler consistently led to performance drops, highlighting its role in capturing valuable context. The Perceiver encoder proved vital for both efficiency and preventing overfitting on smaller datasets. Furthermore, RGP’s multi-task decoder showed that a single model could achieve comparable or even improved results when trained on multiple tasks simultaneously, showcasing its potential for large-scale foundation models in relational domains.
The Relational Graph Perceiver represents a significant step forward in relational deep learning, offering a general, scalable, and efficient solution for integrating complex temporal and structural information across diverse predictive tasks. You can read the full research paper here.
