TLDR: SCoRE is a novel, efficient system for extracting relationships from text, especially in noisy, low-supervision environments. It uses multi-label contrastive learning and Bayesian kNN without fine-tuning large language models, significantly reducing computational costs and energy consumption while maintaining high accuracy and better aligning with knowledge graph structures.
Knowledge Graphs (KGs) are fundamental for organizing structured information, powering applications from search engines to complex question-answering systems. A crucial task in keeping these KGs up-to-date is Relation Extraction (RE), which involves identifying and categorizing the relationships between entities mentioned in text. For instance, given the sentence “Aspirin is commonly prescribed to reduce the risk of heart attacks,” an RE system would predict the “prevent” relation between “Aspirin” and “heart attacks” to enrich a medical KG.
A significant challenge in RE is the scarcity of high-quality, manually annotated data. Distant supervision (DS) attempts to solve this by automatically generating labels from existing KGs, but this often introduces noise, as automatic labels may not always perfectly align with the text’s context. Current RE research frequently relies on deep learning and pre-trained large language models (PLMs), often involving complex fine-tuning strategies to mitigate this noise. However, such methods can be computationally intensive, especially for larger PLMs, and may struggle with adaptability to diverse datasets.
Introducing SCoRE: A New Paradigm for Relation Extraction
A new system called SCoRE (Streamlined Corpus-based Relation Extraction) has been introduced to address these challenges. SCoRE is designed to be a modular, cost-effective, and adaptable sentence-level RE system that integrates seamlessly with PLMs without requiring any fine-tuning. This unique approach allows for easy switching between different PLMs and smooth adaptation to various corpora and KGs.
SCoRE combines supervised contrastive learning with a Bayesian k-Nearest Neighbors (kNN) classifier for multi-label classification. In the training phase, it uses a PLM (like BERT) to encode head and tail entity mentions within sentences into hidden vector representations. These representations are then processed by a small Multi-Layer Perceptron (MLP) and mapped onto a hypersphere using multi-label supervised contrastive learning. This process effectively clusters samples with similar relational patterns together.
During inference, SCoRE leverages a non-parametric multi-label Bayesian kNN approach. It finds the k-nearest neighbors of a new entity pair’s encoding in the learned hidden feature space to predict its relation types. By avoiding PLM fine-tuning, SCoRE treats the PLM as an “informed prior” for encoding, which helps prevent overfitting and reduces computational overhead, making it highly energy-efficient.
Novel Metrics and Real-World Evaluation
To enhance RE evaluation, the researchers propose two novel metrics: Correlation Structure Distance (CSD) and Precision at R (P@R). CSD measures how well the learned relational patterns align with the underlying KG structures, providing insight into the model’s robustness. P@R assesses the system’s utility as a recommender, evaluating the accuracy of its top-ranked predictions.
The paper also introduces Wiki20d, a new benchmark dataset that simulates real-world RE conditions where only KG-derived annotations are available, making it a Fully Distantly Supervised (FDS) corpus. This dataset helps in a more realistic assessment of RE solutions.
Also Read:
- SymbolicThought: Enhancing Narrative Understanding with AI and Logic
- OpenFActScore: Advancing Open-Source Evaluation for AI Text Factuality
Performance and Efficiency
Experiments conducted on five benchmarks, including the new Wiki20d dataset, demonstrate that SCoRE consistently matches or surpasses state-of-the-art methods while significantly reducing energy consumption. For instance, SCoRE’s energy usage during training and testing is remarkably low, often three orders of magnitude less than some competitors, highlighting its potential for sustainable and cost-effective real-world applications.
Further analysis revealed that increasing model complexity, as seen in some prior work, can actually degrade performance when PLM fine-tuning is avoided. This underscores the advantages of SCoRE’s minimal design. SCoRE also showed superior alignment with true relational structures (lower CSD values) compared to other models, indicating its ability to better preserve the logical structure of knowledge graphs.
The research emphasizes that metrics like P@R are more indicative of real-world utility than traditional F1-scores, especially when RE systems act as recommenders. SCoRE’s strong P@R results confirm its effectiveness in such contexts.
In conclusion, SCoRE offers a lightweight, adaptable, and interpretable solution for relation extraction, particularly effective in noisy, low-supervision environments. Its innovative design, which avoids PLM fine-tuning, not only delivers robust performance but also significantly enhances energy efficiency and scalability for real-world RE applications. For more details, you can refer to the full research paper.
