TLDR: A research paper demonstrates that fine-tuning Large Language Models (LLMs) on synthetic code-switched text (mixing languages like Hindi and English) significantly improves their performance in low-resource languages for common sense reasoning tasks, without negatively impacting high-resource language performance. The study found that a moderate level of code-mixing yielded the best results, suggesting a balanced approach to multilingual LLM training.
Large Language Models, or LLMs, have become indispensable tools for communication and understanding across various languages. However, a significant challenge persists: these advanced models often struggle with Common Sense Reasoning (CSR) tasks when prompted in low-resource languages (LRLs) like Hindi or Swahili, performing noticeably worse than in high-resource languages (HRLs) such as English. This disparity limits equitable access to quality LLM outputs for speakers of LRLs and diverse linguistic communities.
A recent research paper titled “Breaking Language Barriers: Equitable Performance in Multilingual Language Models” by Tanay Nagar, Grigorii Khvatskii, Anna Sokol, and Nitesh V. Chawla addresses this critical issue. The authors propose an innovative approach to bridge this performance gap, aiming to ensure fairness and broader utility of LLMs globally. You can find the full research paper at https://arxiv.org/pdf/2508.12662.
The Problem: A Lingering Language Gap
The core issue stems from an imbalance in the training data available for different languages. LLMs are predominantly trained on vast amounts of English text, leading to a performance gap of over 15% on average in CSR tasks across different languages. This imbalance can worsen the digital divide, restricting the benefits of AI advancements for underrepresented communities.
Existing multilingual LLMs often either prioritize a single dominant language or maintain separate internal representations for different languages. This can introduce linguistic biases, especially in CSR tasks that rely on shared human knowledge, potentially skewing the model’s interpretation of diverse cultural contexts.
A Novel Solution: Fine-tuning with Synthetic Code-Switched Text
The researchers propose a method that involves fine-tuning an LLM on synthetic code-switched text. Code-switching is a natural phenomenon where bilingual individuals alternate between two or more languages within the same conversation or even sentence. The paper argues that this practice can lead to a more equitable representation of both HRLs and LRLs, fostering a unified understanding of knowledge across languages within LLMs.
How They Did It: Generating Code-Switched Data
To create the necessary training data, the team employed two main approaches for generating synthetic code-switched text: using a large pre-trained LLM (GPT-3.5) and the CoCoa model. While GPT-3.5 could generate Hinglish (a mix of Hindi and English), it struggled with precise control over language-mixing ratios, often producing inconsistent outputs.
The CoCoa model, however, offered fine-grained control over the Code-Mixing Index (CMI), a measure of how much languages are mixed in a text. CMI ranges from 0% (monolingual) to 50% (maximum code-switching, equal mix). The researchers generated datasets with low (0-16.7%), medium (16.7-30%), and high (30-50%) CMI ranges to study the impact of different mixing intensities.
The CommonSenseQA dataset, a widely accepted benchmark for commonsense reasoning, was chosen as the base. The English questions were transformed into code-switched Hindi-English text, while the answer choices remained in English. This strategy aimed to leverage the model’s strong English understanding and transfer it to Hindi through fine-tuning.
The Experiment and Promising Results
The LLaMA-3-8B-Instruct model was selected as the base for fine-tuning due to its multilingual capabilities and support for both Devanagari (Hindi) and Latin (English) scripts. The model was fine-tuned over five epochs using the generated code-switched datasets.
The results were significant: fine-tuning the LLM on synthetic code-switched datasets substantially improved its performance on Hindi tasks. Crucially, this improvement did not degrade performance on English tasks; in some cases, it even enhanced it. The model fine-tuned with the CMI 2 dataset (medium code-mixing) showed the best performance, achieving an average accuracy of 90.40% on English and 85.60% on Hindi tasks.
This “sweet spot” of moderate code-mixing mirrors observations in human bilingualism, where moderate levels of bilingualism can improve native language performance. Linguistically, moderate code-switching reflects natural bilingual discourse, allowing the model to capture nuanced syntactic structures and semantic relationships across both languages.
Also Read:
- How Language Models Learn Across Languages: The Role of Unified Representations
- Optimizing Data Mixtures for Fine-Tuning Large Language Models
Looking Ahead: Expanding the Impact
While the current study focused on Hindi-English, the researchers acknowledge limitations and plan to expand their experiments to include other low-resource languages and diverse linguistic families, such as Russian-English and Spanish-English. Future work will also explore using more modern text generation techniques like GPT-4o, extending the methodology to other LLMs like Qwen 2.5-7B and Phi-3.5-mini, and evaluating on additional benchmarks like XCOPA and OpenBookQA.
The team also intends to incorporate real code-switched datasets and benchmark their approach against models fine-tuned on fully translated monolingual datasets to further understand the specific effects of code-switching. This research marks a promising step towards building more equitable and high-performing multilingual LLMs for a truly global audience.
