TLDR: A new benchmark, DIVERS-Bench, evaluates language identification (LID) models on diverse, real-world data, including noisy, informal, and code-switched text. The study reveals that while current LID models perform well on clean data, their accuracy significantly degrades on these complex inputs, especially when identifying multiple languages in code-switched sentences or handling low-resource languages. This highlights the critical need for more robust and inclusive LID systems.
Language Identification (LID) is a fundamental task in natural language processing (NLP), crucial for applications like translation, speech recognition, and content moderation. However, a new study introduces DIVERS-Bench, a comprehensive evaluation framework that reveals a significant gap: while current LID systems perform exceptionally well on clean, well-structured data, their accuracy plummets when faced with the complexities of real-world language use.
The research, conducted by Jessica Ojo, Zina Kamel, and David Ifeoluwa Adelani, highlights that existing models often overfit to curated, monolingual datasets. This leads to poor performance on noisy, informal, and especially code-switched inputs—where multiple languages are used within a single sentence. The findings underscore the urgent need for more robust and inclusive LID systems that can truly handle the linguistic diversity of everyday communication.
Introducing DIVERS-Bench: A Holistic Evaluation
DIVERS-Bench is designed to provide a more realistic assessment of LID models by evaluating them across three critical dimensions: domain diversity, language resource imbalance, and code-switched inputs. The benchmark is divided into two main components: DIVERS-MONO for monolingual data and DIVERS-CS for code-switched text.
DIVERS-MONO includes a variety of datasets to simulate different real-world scenarios:
-
FLORES-200: Professionally translated sentences across 200 languages, serving as a clean, standardized benchmark, particularly for low-resource languages.
-
SMOL: A multilingual benchmark with professionally translated data for 115 low-resource languages, incorporating diverse domains.
-
TweetLID: Annotated Twitter data, rich in abbreviations, informal expressions, and messy text, designed to test robustness against noise.
-
Bloom stories: Children’s stories translated into over 100 languages, offering high-quality but simpler text with natural storytelling patterns.
-
Common Voice: A multilingual read-speech dataset, focusing on sentences collected from various language communities.
DIVERS-CS, on the other hand, is a diverse code-switching benchmark covering 10 language pairs, including both high-resource (e.g., English–Spanish) and low-resource (e.g., English–Malayalam) combinations. This is the most comprehensive code-switching dataset to date, specifically curated to include naturally occurring intra-sentential switches with clear language-pair annotations, avoiding synthetic or weakly labeled data.
Models and Metrics
The study evaluated eight state-of-the-art LID models, encompassing a broad range of techniques from traditional statistical methods to modern deep learning approaches. These included Google’s Compact Language Detector 3 (CLD3), FastText LID, Franc, LangDetect, LangID.py, GlotLID, OpenLID, and ConLID. ConLID, in particular, uses supervised contrastive learning to enhance robustness for out-of-domain and low-resource languages.
To measure performance on code-switched text, two key metrics were used: Full Match (FM), which assesses if both languages in a code-switched sentence are correctly identified, and Partial Match (PM), which checks if at least one of the two languages is correctly predicted.
Key Findings: The Reality Check
The evaluation yielded several critical insights:
-
Performance Degradation: Models that performed well on formal data like FLORES-200 showed sharp declines in accuracy on informal datasets such as SMOL and TweetLID. For instance, ConLID’s F1 score dropped from 94.1% on FLORES to 61.6% on SMOL and 59.2% on TweetLID, confirming that noisy, real-world inputs remain a major hurdle.
-
Generalization Limitations: While advanced models like ConLID and GlotLID generally outperformed traditional approaches, they still struggled to generalize across diverse domains, despite being designed for robustness in out-of-domain and low-resource settings.
-
Language Coverage Trade-offs: Models with narrower language coverage showed dramatic improvements in performance when evaluated only on the languages they supported. However, this highlights a trade-off, as broad-coverage models, while showing smaller gains, better reflect real-world utility across many languages.
-
Low-Resource Language Challenges: Performance consistently dropped for low-resource languages (resource classes 0–2), often falling below 30% F1, compared to over 85% for high-resource languages. This indicates that LID accuracy is heavily dependent on the availability of language resources.
-
Code-Switching Struggles: Models performed very poorly on the Full Match (FM) metric for code-switched inputs, with most scores near zero across all language pairs. Even the highest FM score was only 16.0% (OpenLID on eu-es). While Partial Match scores were higher (around 48–50%), indicating models often detect at least one language, they frequently fail to identify both, showing a strong bias towards dominant languages like English. This points to a fundamental limitation of softmax-based LID models in handling intra-sentential multilingual inputs.
Also Read:
- DivLogicEval: A New Benchmark Uncovers Gaps in LLM Logical Reasoning
- EngiBench: A New Standard for Assessing AI in Engineering Challenges
Conclusion and Future Directions
The study concludes that despite advancements, current LID models are not yet equipped to handle the complexities of real-world multilingual text, particularly noisy, informal, and code-switched inputs. The persistent challenges identified by earlier research continue to affect modern systems. The authors recommend training models on data that spans multiple domains, including realistic and noisy text, to improve generalization. They also hope this work encourages evaluation beyond clean benchmarks and drives the development of more robust LID models that truly reflect how people use language in the real world.
For a deeper dive into the methodology and detailed results, you can read the full research paper here.
