TLDR: ESNERA is a novel method for merging Named Entity Recognition (NER) datasets by automatically aligning labels based on their empirical and semantic similarities. It addresses the challenge of inconsistent and small-scale datasets, which are costly to build. ESNERA uses a greedy pairwise merging strategy and parameter optimization to unify label spaces, enhancing NER model performance, especially in low-resource domains. Experiments show it effectively merges more labels than manual methods while maintaining high accuracy, demonstrating its scalability and interpretability.
Named Entity Recognition (NER) is a cornerstone task in natural language processing, crucial for applications ranging from information extraction to building knowledge graphs. While deep learning has significantly advanced NER performance, its success heavily relies on the availability of large, high-quality annotated datasets. However, creating these datasets is a costly and time-consuming endeavor, presenting a major hurdle for further research and practical application.
Current methods for merging NER datasets often fall short. Approaches like manual label mapping or constructing label graphs require extensive expert involvement and lack the ability to scale efficiently. Other techniques, such as pseudo-labeling, can introduce noise and are difficult to interpret, making it challenging to ensure accurate label alignment across diverse datasets.
Addressing these limitations, a new method called ESNERA (Empirical and Semantic Named Entity Alignment) has been proposed. ESNERA offers an automatic, interpretable, and scalable solution for unifying label spaces across different NER datasets. Its core innovation lies in combining both empirical and semantic similarities to align named entity labels.
How ESNERA Works
ESNERA operates by calculating two complementary types of similarity between labels from different datasets:
- Empirical Similarity: This measures how well an NER model, trained on one dataset, predicts labels on another. It essentially quantifies the overlap in annotation practices between datasets. For example, if a model trained on a dataset where ‘address’ is a label frequently predicts ‘GPE’ (Geopolitical Entity) in another dataset, it suggests a strong empirical similarity between ‘address’ and ‘GPE’. This similarity is directional, meaning the prediction from dataset A to B might differ from B to A.
- Semantic Similarity: This assesses the conceptual closeness of labels by analyzing the contextual meaning of the entities they represent. ESNERA uses advanced language models like BERT to generate dynamic word embeddings for entities. These embeddings capture the semantic information of words based on their context. By averaging and normalizing these embeddings for all entities under a specific label, ESNERA can calculate how semantically close two labels are, regardless of their annotation frequency.
These two similarity measures are then combined using a linear interpolation approach, allowing ESNERA to balance the influence of annotation patterns and semantic meaning. A key aspect of ESNERA is its greedy pairwise merging strategy. Instead of attempting a complex global merge, it aligns two datasets at a time, creating an intermediate dataset that then merges with the next. This simplifies the process and reduces potential errors. The method also employs a grid search to find optimal parameters that maximize the number of merged labels while ensuring the NER model’s performance remains stable.
Furthermore, ESNERA includes a label augmentation step. If a label type exists in a source dataset but is missing in a target dataset, ESNERA uses pseudo-labeling to fill in these gaps. This ensures a complete label space in the merged dataset, which is vital for robust multi-source NER training.
Experimental Validation
The effectiveness of ESNERA was tested on several Chinese NER datasets, including OntoNotes, CLUENER, and BosonNER, which vary significantly in entity types, annotation styles, and domain backgrounds. The experiments showed that ESNERA successfully merged 15 labels, outperforming manual merging methods which only unified 11 labels, all while maintaining a competitive micro-averaged F1 score for NER performance. This indicates ESNERA’s ability to expand label coverage without sacrificing accuracy.
In an alternative scenario, ESNERA was applied to a small, resource-limited financial dataset called FinReportNER. By leveraging a larger source dataset, ESNERA improved the NER performance on FinReportNER, demonstrating its cross-domain transferability and robustness in low-resource situations. For instance, the ‘ORG’ (Organization) label saw significant improvement due to the merging of related terms like ‘company’ and ‘government’.
Ablation studies confirmed that both empirical and semantic similarity components are crucial for ESNERA’s comprehensive and scalable label merging capabilities. The findings highlight that relying on only one type of similarity leads to fewer successful merges.
Also Read:
- Enhancing Search Accuracy with Coreference-Linked Passage Augmentation
- Decoding Data Flow: How AI is Improving Schema Understanding
Conclusion
ESNERA presents a significant step forward in addressing the challenges of named entity dataset merging. By systematically modeling label relationships through combined empirical and semantic similarities, it offers an efficient, interpretable, and scalable solution for integrating multi-source NER corpora. This work not only enhances label coverage but also preserves or improves the recognition performance of most labels, particularly benefiting from additional training data. While some challenges remain, especially for labels with very limited samples or those prone to semantic drift, ESNERA provides a strong foundation for future advancements in cross-domain and potentially multilingual NER tasks. For more in-depth details, you can refer to the full research paper: ESNERA: Empirical and semantic named entity alignment for named entity dataset merging.
