TLDR: LUXINSTRUCT is a new, high-quality cross-lingual instruction tuning dataset for Luxembourgish. It addresses the scarcity of data for low-resource languages by leveraging aligned data from English, French, and German, without relying on machine translation into Luxembourgish. Experiments show that this approach significantly improves LLM performance, enhancing cross-lingual alignment and generative capabilities in Luxembourgish, offering a robust foundation for future NLP development in the language.
Large Language Models (LLMs) have transformed how we interact with technology, making them better at understanding and responding to human instructions. However, this progress has largely benefited high-resource languages like English, leaving many low-resource languages, such as Luxembourgish, struggling to catch up.
The main hurdle for these languages is the lack of high-quality datasets that teach LLMs how to follow instructions. Traditionally, researchers have tried to overcome this by using machine translation (MT) to convert existing datasets into the target low-resource language. But this approach often falls short. Machine translations can miss subtle meanings, cultural nuances, and common expressions, leading to responses that feel unnatural or even incorrect. This can severely impact an LLM’s performance.
Introducing LUXINSTRUCT: A New Approach for Luxembourgish
A new research paper introduces LUXINSTRUCT, a groundbreaking cross-lingual instruction tuning dataset specifically designed for Luxembourgish. What makes LUXINSTRUCT unique is its innovative approach: it avoids machine-generated translations directly into Luxembourgish. Instead, it cleverly uses aligned data from English, French, and German to build a high-quality dataset that respects the linguistic and cultural specificities of Luxembourgish.
The creators of LUXINSTRUCT believe that this method not only helps align the way LLMs represent different languages but also significantly boosts their ability to generate text in Luxembourgish. This highlights a crucial point: carefully curated cross-lingual data can sidestep the common problems of machine-translated data and directly support the development of LLMs for languages with fewer digital resources.
How LUXINSTRUCT Was Created
The dataset was built using three main sources:
Wikipedia: The team used Luxembourgish Wikipedia articles to generate English instructions, with the outputs remaining in Luxembourgish. This ‘reverse instruction generation’ method ensures high-quality, semantically aligned instruction-output pairs without relying on machine translation into Luxembourgish.
News Articles: Leveraging RTL Luxembourg, a news platform publishing in Luxembourgish, French, and English, the researchers identified bilingual article pairs. From these, they created tasks like generating Luxembourgish news headlines from English or French articles, or vice-versa.
Online Dictionary: A publicly available Luxembourgish dictionary provided lexical entries, synonyms, translations, and example sentences. This source was used to create tasks such as generating example sentences for Luxembourgish words, simplifying colloquial Luxembourgish sentences, and word translation.
The final LUXINSTRUCT dataset contains over 390,000 cross-lingual instruction-output samples (with instructions in English, French, or German and outputs in Luxembourgish) and nearly 146,000 monolingual samples (both instruction and output in Luxembourgish).
Also Read:
- Enhancing Cross-Lingual Abilities in LLMs with Code-Switching
- SynCED-EnDe: A New Benchmark for Safer English-German Machine Translation
Demonstrating the Benefits
The researchers conducted experiments to compare cross-lingual and monolingual instruction tuning. Their findings clearly show that cross-lingual instruction tuning is highly effective, often outperforming monolingual methods. It significantly improves the alignment between Luxembourgish and other language representation spaces within LLMs. Furthermore, using cross-lingual examples in few-shot in-context learning (where the model learns from a few examples) led to better text generation capabilities in Luxembourgish compared to using only monolingual examples.
This work represents a major step forward for Luxembourgish language technology. By providing human-written outputs, LUXINSTRUCT ensures natural and reliable targets for LLMs, improving their linguistic accuracy, grammar, and cultural understanding in Luxembourgish. The dataset draws on local knowledge from Wikipedia and current events from news articles, embedding a context-aware understanding of the language within LLMs.
While the current dataset is a strong foundation, the team plans to expand it further in size and diversity, applying it to larger LLMs to enrich their Luxembourgish capabilities even more. You can read the full research paper for more details here: LUXINSTRUCT Research Paper.
