TLDR: A new research paper introduces CorIL, a large-scale, high-quality parallel corpus designed to improve machine translation for Indian languages. Covering 11 languages and three key domains (Government, Health, General), CorIL features 772,000 sentence pairs, with a significant portion sourced natively to preserve linguistic authenticity. Experiments with models like IndicTrans2, NLLB, and BhashaVerse show that fine-tuning with CorIL leads to performance improvements, especially with bilingual strategies, while also highlighting challenges in domain adaptation and low-resource languages.
India is a land of incredible linguistic diversity, home to over 120 major languages and approximately 1,600 additional languages, with 22 officially recognized by the Constitution. While advancements in neural machine translation (NMT) have been significant, a persistent challenge remains: the scarcity of high-quality parallel corpora, especially for Indian languages across various specialized domains.
A new research paper, titled “CorIL: Towards Enriching Indian Language to Indian Language Parallel Corpora and Machine Translation Systems,” addresses this critical gap by introducing CorIL, a substantial and meticulously curated parallel corpus. This dataset aims to significantly enhance the availability of high-quality training data for Indian languages, providing a valuable resource for the machine translation research community. You can read the full paper here.
Introducing CorIL: A Comprehensive Linguistic Resource
CorIL is a large-scale, high-quality annotated parallel corpus that spans 11 languages: English, Telugu, Hindi, Punjabi, Odia, Kashmiri, Sindhi, Dogri, Kannada, Urdu, and Gujarati. It comprises a total of 772,000 bi-text sentence pairs. What makes CorIL particularly valuable is its careful curation and systematic categorization into three crucial domains: Government, Health, and General. This domain-specific approach is designed to facilitate domain-aware machine translation research and enable effective adaptation of models to different contexts.
A key innovation of CorIL is its focus on linguistic authenticity. Unlike many existing corpora that rely on translations from English, CorIL enhances the landscape by sourcing texts originally written in low-resource Indian languages and then translating them into both Hindi and English. This method helps preserve linguistic nuances and cultural context, which are often lost when the source text is not native.
Languages and Their Unique Challenges
The dataset covers languages from major families like Indo-Aryan (Dogri, Hindi, Gujarati, Kashmiri, Oriya, Punjabi, Urdu) and Dravidian (Telugu, Kannada), alongside English. These languages present unique challenges for machine translation due to their morphological richness and diverse script systems. For instance, while most Indian languages use Brahmi-derived scripts, Urdu and Sindhi utilize the Perso-Arabic script, adding another layer of complexity.
- Sindhi: Spoken by millions, it uses both Perso-Arabic and Devanagari scripts, posing orthographic challenges.
- Gujarati: Spoken by over 55 million, it uses its own script derived from Devanagari, featuring rich inflectional morphology.
- Dogri: An Indo-Aryan language primarily in Jammu and Kashmir, it was officially recognized in 2003 and is written in Devanagari.
- Kannada: A major Dravidian language with over 43 million speakers, it has a rich literary tradition and agglutinative morphology.
- Telugu: The fourth most spoken language in India with over 81 million speakers, it is a Dravidian language with a script closely related to Kannada.
- Punjabi: A tonal Indo-Aryan language spoken by over 125 million globally, it uses Gurmukhi in India and Shahmukhi in Pakistan.
- Odia: An Indo-Aryan classical language with over 40 million speakers, it uses its unique rounded script.
- Urdu: An Indo-Aryan language primarily in Pakistan and parts of India, it evolved during the Mughal era and is written in the Persian script.
- Kashmiri: An Indo-Aryan language from the Dardic group, spoken by around 7 million, it uses both Perso-Arabic and Devanagari scripts and has an ergative-absolutive syntactic structure.
- Hindi: One of the most widely spoken languages globally, it is an official language of India, written in Devanagari, and draws heavily from Sanskrit.
- English: A global lingua franca, serving as a point of comparison and a pivot language in translation efforts.
Domains of Focus
The CorIL corpus is strategically divided into three domains to cater to real-world translation needs:
- Governance: Includes official records, policies, and public service announcements. Language technology in this domain supports administrative efficiency and citizen engagement.
- Healthcare: Crucial for improving medical communication, patient care, and health information accessibility, especially in bridging communication gaps between providers and patients.
- General: Comprises everyday communication, news, and informal discourse, reflecting natural language use and enhancing a system’s adaptability to diverse user interactions.
Evaluating Machine Translation Models with CorIL
To demonstrate CorIL’s utility, the researchers fine-tuned and evaluated several state-of-the-art NMT models, including IndicTrans2, NLLB (No Language Left Behind), and BhashaVerse. These models were tested using both bilingual (two-language specific) and multilingual (many-language) fine-tuning strategies across the different domains.
The analysis revealed several important performance trends. Fine-tuning with CorIL consistently led to improvements over the baseline performance of all models on the domain-specific test sets. However, a contrasting trend emerged when these fine-tuned models were evaluated on public benchmark datasets like FLORES-200 and IN22 general test sets, where a decline in performance suggested potential domain overfitting.
Performance also varied significantly based on the language script. Massively multilingual models like NLLB and BhashaVerse showed an advantage on Perso-Arabic scripts (Urdu, Sindhi), while IndicTrans2 excelled on Indic scripts (like Punjabi). This highlights the importance of model architecture and pre-training data in handling diverse linguistic characteristics. Low-resource languages such as Kashmiri, Sindhi, and Dogri consistently yielded lower scores, underscoring the ongoing challenge of data scarcity for these languages.
Interestingly, the government domain generally proved easier to translate than the healthcare domain, likely due to the more standardized language used in governance texts compared to the highly specialized terminology in healthcare. Bilingual fine-tuning generally proved to be a more effective and reliable strategy across most language pairs and domains compared to multilingual fine-tuning.
Also Read:
- Advancing English–Tigrinya Machine Translation with Custom Tokenizers and Refined Evaluation
- Navigating the Adequacy-Fluency Tradeoff in Machine Translation Evaluation
Conclusion and Future Impact
The introduction of CorIL marks a significant step forward in enriching parallel corpora for Indian languages. By providing a high-quality, domain-specific dataset that includes underrepresented languages, this work addresses a critical need in the machine translation community. The empirical studies conducted with CorIL offer valuable insights into model capabilities, domain sensitivity, and cross-script transfer learning, paving the way for more impactful and inclusive research in machine translation for India’s diverse linguistic landscape.
