TLDR: This research challenges the common belief that Large Language Models (LLMs) perform worse in non-source languages due to a lack of knowledge. Instead, the authors propose that the ‘cross-lingual gap’ is primarily caused by increased *variance* in LLM responses when queried in target languages. They demonstrate that simple techniques like ensembling multiple responses or providing multiple translations in the prompt can significantly reduce this variance and improve accuracy across languages, suggesting that the knowledge is present but inconsistently retrieved.
Large Language Models (LLMs) have transformed how we access information, aiming to make knowledge universally available across many languages. Modern LLMs, like Gemini and GPT-5, support dozens of languages, yet a persistent challenge known as the ‘cross-lingual gap’ remains. This gap refers to a noticeable drop in accuracy when an LLM is asked a question in a target language compared to when the same question is posed in its source language.
Traditionally, this performance difference has been attributed to ‘knowledge fragmentation’ or ‘representation misalignment’ – the idea that an LLM might not embed information about an entity (like ‘Nelson Mandela’) consistently across different languages or scripts. This perspective suggests that the model simply lacks the knowledge in the target language.
However, a new research paper titled ‘RETHINKINGCROSS-LINGUALGAPS FROM ASTATISTICALVIEWPOINT’ by Vihari Piratla, Purvam Jain, Darshan Singh, Partha Talukdar from Google DeepMind, and Trevor Cohn from Google Research, offers an alternative explanation. This work hypothesizes that the primary cause of the cross-lingual gap is not a lack of knowledge, but rather the increased *variance* in the LLM’s responses when queried in a target language.
To understand this, imagine asking an LLM ‘When was Kreeda Bharti established?’ with the correct answer being ‘1992’. If the gap were due to a knowledge barrier (bias), the model might give a wildly incorrect answer, like ‘1500 BC’. But if the gap is due to variance, the model might give answers like ‘1990’, ‘1995’, or ‘1992 ± 30’ – answers that are distributed around the correct one, but with more spread. The researchers found that, in practice, target language responses often cluster around the correct answer, indicating high variance rather than a fundamental lack of knowledge.
The paper formalizes this cross-lingual gap using a bias-variance decomposition, a statistical tool for understanding error. Their extensive experiments provide strong evidence for the variance hypothesis. They demonstrated that simple interventions at inference time, designed to control response variance, significantly reduce the cross-lingual gap.
One key method explored was ‘response ensembling’, where the model is prompted multiple times for the same question, and the most frequent or ‘majority voted’ response is chosen. This technique consistently reduced the divergence between source and target language responses across various LLMs and benchmarks (ECLeKTic and MMLU with mixup). For example, ensembling responses improved target accuracy by 20-25% in some cases.
Another approach, ‘input ensembling’, involved presenting the original question along with several translations in a single prompt (Translation Ensemble, TrEn-k) or asking the model to generate translations before answering (Translate-then-Answer, TTA). These methods also led to improved cross-lingual transfer scores, further supporting the idea that the knowledge is present but needs to be elicited more consistently.
A surprising finding was that the variance in source and target responses are proportional. This means that when an LLM is highly confident and consistent in its responses in the source language, the cross-lingual gap tends to diminish. This suggests that improving the model’s certainty in its primary language can have positive ripple effects on its multilingual performance.
The implications of this research are significant. Instead of requiring expensive re-training or fundamental changes to LLM architectures to address knowledge fragmentation, the findings suggest that simpler, post-training or inference-time strategies focused on reducing response variance can be highly effective. This could lead to more efficient ways to improve the multilingual capabilities of LLMs.
Also Read:
- ParallaxRAG: A Multi-View Approach to Enhance LLM Reasoning with Knowledge Graphs
- Evaluating Language Models on Real-World Uncertainty with OPENESTIMATE
For more details, you can read the full research paper: Rethinking Cross-Lingual Gaps from a Statistical Viewpoint.
