TLDR: A new method called TRepLiNa, combining Centered Kernel Alignment (CKA) and REPINA regularization, significantly improves machine translation quality for low-resource languages (LRLs) when used with the Aya-23 8B multilingual language model. By aligning internal representations in mid-level layers and stabilizing high-resource language features, TRepLiNa offers a practical and effective solution for data-scarce translation challenges.
Many of the world’s languages suffer from a significant digital divide: they are considered ‘low-resource languages’ (LRLs) because there isn’t much digital text or data available for them. This scarcity makes it incredibly challenging to build effective machine translation tools, especially for diverse regions like India, which is home to numerous LRLs. This problem is a key focus of the 2025 Multimodal Models for Low-Resource Contexts and Social Impact (MM-LoSo) Language Challenge.
A recent research paper introduces a novel approach called TRepLiNa, which aims to bridge this gap. The study investigates whether enhancing cross-lingual similarity within specific internal layers of a decoder-only multilingual large language model (LLM) can boost translation quality from LRLs to high-resource languages (HRLs).
The Challenge with Multilingual LLMs
Large multilingual language models, such as Aya-23 8B, are designed to understand and process many languages. However, their performance often lags when dealing with LRLs compared to HRLs. Previous observations on Aya-23 8B indicated that while the initial layers of the model show strong overlap in how they represent related languages, this similarity tends to diminish in the intermediate and higher layers. This suggested a clear hypothesis: if we could selectively increase cross-lingual similarity where it’s weakest – in the mid-to-high layers – we might achieve better translation for LRLs.
Introducing TRepLiNa: CKA + REPINA
To address this, the researchers developed TRepLiNa, a method that combines two powerful techniques:
- Centered Kernel Alignment (CKA): This is a metric used to measure and encourage similarity between the internal representations of different languages within the model. Essentially, it helps the model learn to process similar meanings in different languages in a consistent way.
- REPINA: This is a regularization method that acts as a stabilizer. It prevents the representations of high-resource languages from drifting or becoming distorted during the alignment process. It ensures that while LRLs are being aligned, the HRLs maintain their quality and stability.
TRepLiNa works by applying CKA to pull the LRL representations closer to the HRL representations at a chosen layer, while REPINA simultaneously anchors and stabilizes the HRL features. This targeted alignment aims to improve LRL translation without degrading the quality of HRL processing.
Experiments and Key Findings
The study conducted experiments using the MM-LoSo shared task dataset, focusing on translation from LRLs (Mundari, Santali, Bhili) to HRLs (Hindi, English). They tested TRepLiNa in various settings, including zero-shot (no examples), few-shot (a few examples), and fine-tuning with QLoRA on Aya-23 8B.
The results were compelling:
- Optimal Layer Alignment: TRepLiNa showed the most effectiveness when aligning mid-level layers, specifically around layer 15. This suggests a ‘sweet spot’ in the model’s architecture where alignment yields the best improvements.
- Significant Performance Gains: For language pairs like Mundari-Hindi and Santali-English, TRepLiNa consistently achieved the best translation scores, significantly outperforming baseline methods, including zero-shot, few-shot, and even using REPINA alone. For Santali, which presents unique challenges due to its distinct script and longer sentence structures, TRepLiNa delivered a substantial improvement over existing benchmarks.
- Language Proximity Matters: The researchers also observed that for closely related language pairs, such as Bhili-Hindi, a very strong CKA weight could sometimes lead to ‘over-alignment,’ potentially erasing beneficial language-specific nuances. In these cases, a slightly reduced CKA weight proved more effective.
- Data Scaling Insights: The study found that when working with larger datasets, a lower CKA weight was preferable. This indicates that as cross-lingual representations naturally align with more data, excessive alignment pressure can become counterproductive.
Also Read:
- TreePrompt: Enhancing Machine Translation Through Intelligent Example Selection
- Boosting Luxembourgish Language Models with a New Cross-Lingual Dataset
Conclusion
The TRepLiNa method, by combining CKA for similarity and REPINA for stability, offers a practical and low-cost solution for improving machine translation in low-resource contexts. It demonstrates that strategically aligning representations in the middle layers of a multilingual LLM can yield robust gains, especially in data-scarce environments. This research provides valuable guidelines on when and where to apply such alignment techniques, paving the way for more inclusive and effective multilingual AI systems. The code for TRepLiNa is publicly available for further research and development. Read the full paper here.
