TLDR: MERLIN is a new dataset and testbed for Multilingual Multimodal Entity Recognition and Linking (MMEL). It combines BBC news article titles with images in five non-English languages (Hindi, Japanese, Indonesian, Vietnamese, Tamil) to link over 7,000 entity mentions to Wikidata. The research shows that visual data significantly improves entity linking accuracy, especially for ambiguous text and models with weaker multilingual capabilities, highlighting the importance of multimodal approaches for diverse languages.
A new research paper introduces MERLIN, a groundbreaking testbed designed to advance Multilingual Multimodal Entity Recognition and Linking (MMEL). This innovative system addresses the complex challenge of identifying and linking ambiguous entity mentions from unstructured data to a knowledge base, especially when dealing with multiple languages and incorporating visual information.
Traditional entity linking primarily focused on text. However, in today’s digital landscape, particularly with short news clips or social media posts, text alone can often be insufficient to accurately disambiguate entities. Images, which frequently accompany text, offer crucial additional context. Furthermore, entity linking in non-English languages presents its own set of challenges due to limited resources.
MERLIN tackles these issues by combining both textual and visual data in a multilingual context. The dataset comprises BBC news article titles, each paired with a corresponding image, across five diverse languages: Hindi, Japanese, Indonesian, Vietnamese, and Tamil. It features over 7,000 named entity mentions linked to 2,500 unique Wikidata entities. This makes MERLIN the first testbed of its kind, pushing the boundaries of entity linking into a truly multilingual and multimodal setting.
The creation of the MERLIN dataset involved a meticulous process. Languages were selected based on linguistic diversity, speaker population, and annotator availability. Annotators, recruited through the Prolific Crowdsourcing Platform, underwent a rigorous two-stage screening process to ensure high-quality annotations. The INCEpTION tool was used for annotation, allowing for linking text to Wikidata, the chosen knowledge base.
The researchers evaluated several existing multilingual and multimodal entity linking methods on MERLIN, including mGENRE and GEMEL (with Llama-2 and Aya-23 as text encoders). A key finding was that incorporating visual data significantly improves the accuracy of entity linking, particularly for entities where textual context is ambiguous or insufficient. This benefit was especially pronounced for models that do not possess strong inherent multilingual abilities, like Llama-2, which showed a notable performance drop when images were removed. Aya-23, being a more multilingual model, was less dependent on visual input but still benefited.
The study also highlighted that ambiguous mentions—where the same textual mention can refer to multiple distinct Wikidata entities—are particularly challenging. While visual inputs offered some gains, performance remained lower for these cases, indicating an area for future research. Person mentions generally benefited most from visual context, while organization mentions were easier to resolve. Miscellaneous and event mentions proved consistently challenging.
Interestingly, translating non-English data into English using the NLLB model generally led to an improvement in performance across all languages, underscoring the existing disparity in understanding between English and non-English contexts in current methods. This suggests that while multimodal approaches are vital, further advancements are needed to enhance entity linking accuracy in diverse linguistic environments.
Also Read:
- New Benchmark Reveals Gaps in Vision Language Model Understanding Across Diverse Languages
- Bridging the Language Gap: LiRA Framework Boosts LLM Performance in Low-Resource Languages
MERLIN serves as a valuable resource for the research community, providing a robust benchmark for evaluating future entity linking models. While the dataset currently focuses on BBC news articles, future work could expand its genre diversity. For more details, you can read the full research paper here.
