TLDR: A new study challenges common beliefs in multilingual large language model (LLM) pretraining. Researchers found that increasing English data doesn’t necessarily harm multilingual performance if enough non-English data is present. English acts as a strong pivot language across diverse language families, and curriculum learning doesn’t improve final performance. The ‘curse of multilinguality’ is redefined as a ‘curse of capacity’ or ‘data quality,’ emphasizing that performance degradation stems from finite model capacity and noisy low-resource data, rather than simply adding more languages.
A recent study from EPFL researchers, titled Revisiting Multilingual Data Mixtures in Language Model Pretraining, delves into the complexities of training large language models (LLMs) with diverse multilingual datasets. The research challenges several long-held assumptions about how different language mixtures impact model performance, particularly concerning the so-called ‘curse of multilinguality’ and the role of English as a pivot language.
The team trained LLMs with 1.1 billion and 3 billion parameters on corpora containing anywhere from 25 to 400 languages. Their systematic approach allowed for a detailed examination of how language count, diversity, and token distribution affect model capabilities.
English Data: Friend, Not Foe?
One of the most significant findings challenges the notion that including more English data in a multilingual training mix inevitably harms the performance of other languages. The study found that as long as a sufficient amount of non-English data (multilingual tokens) is also present, increasing the proportion of English data does not degrade performance for either English or the other languages. This suggests that developers can prioritize strong English performance without necessarily sacrificing multilingual capabilities, provided the overall multilingual data volume is adequate.
English as a Universal Pivot
Another key assumption debunked by the research is that cross-lingual transfer is most effective between languages from the same family. Contrary to this belief, the study observed that English consistently acts as a highly effective ‘pivot language,’ providing benefits across various language families. While a family-specific pivot (like Russian for Slavic languages) might offer slight advantages in extremely low-resource scenarios, combining English with such a pivot often yields the best overall results. This highlights English’s unique position, likely due to the vast amount of diverse and high-quality data available for it online.
Curriculum Learning’s Limited Impact
The researchers also investigated whether curriculum learning—a strategy where languages are introduced in a staged order during training—could reduce negative interference between languages and improve performance. Their experiments showed that while curriculum learning can influence the learning trajectory, it does not ultimately mitigate negative interference or lead to better final performance for non-English languages. Any observed gains for English under certain curricula were attributed to the sheer quantity of English data rather than the specific learning order.
Rethinking the ‘Curse of Multilinguality’
Perhaps the most widely discussed concept in multilingual LLM training is the ‘curse of multilinguality,’ which posits that adding more languages beyond a certain point degrades model performance. This study refines this understanding, suggesting that the degradation isn’t simply due to the number of languages. Instead, it arises from the finite capacity of models and data distributions that disproportionately amplify the impact of noisy, low-resource languages. The researchers propose that it’s more accurately described as a ‘curse of capacity’ (when models can’t absorb all the data) or a ‘curse of data quality’ (when oversampling low-resource languages introduces too much noise).
Also Read:
- The Hidden Peril: How English-Only Unlearning Fails Multilingual AI
- Unlocking Multilingual Potential: Prefix-Based Methods for Zero-Shot Language Transfer
Practical Implications for LLM Development
These findings offer valuable guidance for practitioners building multilingual LLMs. Firstly, complex curriculum learning strategies may not be necessary; a well-mixed data approach appears equally effective. Secondly, instead of meticulously balancing data proportions to reduce English, efforts should focus on acquiring and cleaning high-quality data for low-resource languages. Lastly, the study encourages broader language coverage, emphasizing that the quality and quantity of data per language are more critical than the total number of languages in overcoming performance limitations.
While the study utilized 1.1B and 3B parameter models, the authors acknowledge that future work should validate these principles on even larger, frontier-scale models to fully understand how increased capacity might alter these relationships.
