TLDR: A study on the CORE-KG framework, which builds knowledge graphs from legal texts, found that its two main components—type-aware coreference resolution and structured prompts—are both crucial and complementary. Removing coreference resolution increased duplicate nodes by 28.32% and noisy nodes by 4.32%. Removing structured prompts led to a 4.34% increase in duplicate nodes and a significant 73.33% increase in noisy nodes. This highlights that structured prompts are key for extraction precision and noise reduction, while coreference resolution is vital for consolidating entities and resolving ambiguity, both essential for high-quality knowledge graphs.
Analyzing complex human smuggling networks is a critical task for law enforcement and policymakers, but much of the vital intelligence is buried within unstructured legal documents. These texts are often dense, filled with ambiguous references, and pose significant challenges for automated knowledge graph construction. Traditional methods struggle with the inconsistencies of legal language, leading to fragmented graphs with duplicate entries and irrelevant information. Large Language Models (LLMs) offer a promising avenue, but even they can produce noisy and redundant knowledge graphs without proper guidance.
This is where the CORE-KG framework comes into play. Introduced as a modular solution, CORE-KG aims to build clean and interpretable knowledge graphs from legal case documents. It achieves this by integrating two key components: a type-aware coreference resolution module and a domain-guided structured prompting strategy. The coreference module intelligently unifies different mentions of the same entity (e.g., “Defendant Lewis,” “smuggler,” “Lewis”) into a single node, while structured prompts guide the LLM to extract only relevant entities and relationships, filtering out legal jargon and reducing ambiguity.
A recent study, detailed in the paper “Inside CORE-KG: Evaluating Structured Prompting and Coreference Resolution for Knowledge Graphs”, delves into the individual contributions of these two components through a systematic ablation study. The researchers, Dipak Meher and Carlotta Domeniconi from George Mason University, sought to quantify exactly how much each part contributes to the overall quality of the knowledge graphs generated from human smuggling case documents.
The Role of Coreference Resolution
The study found that removing the coreference resolution module significantly impacts graph quality. Without it, there was a substantial 28.32% increase in node duplication. This means that entities like “Interstate Highway 35,” “Interstate,” and “I-35” would be treated as separate nodes instead of being recognized as the same route. While the structured prompts alone did help reduce some duplication compared to a basic LLM approach, coreference resolution proved essential for merging these varied mentions. Additionally, removing coreference resolution led to a 4.32% increase in noisy nodes, suggesting it also plays a role in refining the input text and preventing misclassifications.
The Power of Structured Prompts
The impact of structured prompts was even more pronounced, particularly in controlling noise. The study revealed that removing the structured prompting strategy resulted in a massive 73.33% increase in noisy nodes. This component is crucial for guiding the LLM to extract only information relevant to human smuggling networks, effectively filtering out legal boilerplate terms (like “Court,” “Appeal Process”) and preventing the model from overgeneralizing or misclassifying entities. It also contributed to reducing node duplication by 4.34%, primarily by enforcing sequential extraction and providing clear entity type definitions, which minimizes attention drift and ensures precise entity boundaries.
Also Read:
- Advancing Criminal Network Analysis with AI-Powered Knowledge Graphs
- Navigating Regulatory Compliance with Dual-Graph AI
Complementary Contributions for Robust Knowledge Graphs
The findings clearly demonstrate that both type-aware coreference resolution and domain-guided structured prompts are not just beneficial, but essential and complementary for constructing accurate and interpretable knowledge graphs from complex legal texts. Structured prompts are dominant in improving extraction precision and reducing noise, while coreference resolution excels at enhancing entity consolidation by resolving referential ambiguity. When combined in the full CORE-KG pipeline, they achieve the strongest performance, leading to cleaner, more coherent, and more informative graphs.
This research offers valuable empirical insights for designing future LLM-based pipelines for knowledge graph construction, especially in challenging domains like legal text analysis. By understanding the distinct impacts of these components, developers can create more robust and effective systems for extracting structured representations from unstructured data, ultimately aiding in the analysis and disruption of complex networks like human smuggling operations.
