TLDR: Alif-1.0-8B-Instruct is a new Urdu-English large language model that significantly outperforms existing multilingual LLMs on Urdu-specific tasks. It achieves this by using a novel modified self-instruct technique to create a high-quality, culturally nuanced synthetic Urdu-Instruct dataset. The model, built on Llama-3.1-8B, demonstrates superior performance and ethical alignment for low-resource languages, all while maintaining a remarkably low training cost of under $100.
Developing powerful large language models (LLMs) for languages with fewer digital resources, like Urdu, comes with unique challenges. These include a shortage of high-quality data, inconsistencies when dealing with multiple languages, and safety concerns. Often, existing multilingual LLMs try to solve these problems by translating vast amounts of data from high-resource languages. However, these translations can sometimes miss cultural nuances and quality, and they can be expensive to create and train.
To tackle these issues, a new model called Alif-1.0-8B-Instruct has been introduced. This model is a multilingual Urdu-English LLM that uses a novel approach: it’s trained on a high-quality, multilingual synthetic dataset called Urdu-Instruct. This dataset was created using a special, modified self-instruct technique. This technique uses unique prompts and seed values for each task, along with a global task pool, to incorporate Urdu-native chain-of-thought reasoning, bilingual translation, cultural relevance, and ethical safety alignments. This significantly improves Alif-1.0-8B-Instruct’s understanding of Urdu-specific tasks.
Alif-1.0-8B-Instruct is built upon the pretrained Llama-3.1-8B model. It has shown superior performance compared to Llama-3.1-8B-Instruct for tasks specific to Urdu. It also outperformed other leading multilingual LLMs such as Mistral-7B-Instruct-v0.3, Qwen-2.5-7B-Instruct, and Cohere-Aya-Expanse-8B. Remarkably, this was achieved with a training budget of under $100. The results suggest that high-performance LLMs for low-resource languages can be developed efficiently and with cultural alignment using this modified self-instruct approach. All datasets, models, and code are publicly available.
The rapid progress in LLMs has transformed natural language processing across many languages. However, a significant gap remains between languages with abundant resources, like English, and those with fewer, like Urdu. This disparity creates technological barriers for billions of speakers of underrepresented languages, limiting their access to AI tools. Including low-resource languages in LLM development is vital for creating inclusive, globally accessible AI systems.
Challenges in developing LLMs for low-resource languages include data scarcity, multilingual inconsistencies, translation inaccuracies, reasoning limitations, and ethical concerns. Traditional methods often rely on translated data, which can fail to capture regional knowledge and cultural nuances, leading to less effective communication. Urdu, with its unique alphabet, complex grammar, syntax, and morphology, presents additional difficulties. It also borrows from languages like Hindi, Punjabi, and Persian and is written in both Perso-Arabic and Devanagari scripts.
The Alif-1.0-8B-Instruct model addresses these challenges by using a modified self-instruct technique. This method incorporates a carefully curated Urdu dataset designed to improve Urdu generation quality, bilingual translation, culturally aware understanding, and Urdu-native chain-of-thought reasoning. This unique multilingual synthetic data distillation approach not only boosts the model’s performance on Urdu and English tasks but also maintains ethical commitments to safety and cultural sensitivity.
The training process for Alif-1.0-8B-Instruct involves several stages: continued pretraining to build foundational understanding, fine-tuning on the synthetic Urdu-Instruct dataset for enhanced comprehension, incorporating translated Urdu data for broader knowledge, and replaying English data to prevent catastrophic forgetting. This structured approach ensures robust linguistic capabilities across diverse tasks.
A key contribution is the modified self-instruct technique, which uses diverse prompts and a global task pool. Each task is guided by unique prompts and seed values to capture cultural diversity, output structure, and task-specific nuances. A centralized task pool with human feedback ensures uniqueness and prevents redundancy. This scalable method improves instruction quality and can be adapted to other low-resource languages.
The high-quality Urdu-Instruct dataset, consisting of 51,686 examples generated using GPT-4o, is central to Alif’s success. It covers seven key Urdu tasks: Generation, Ethics, QA, Reasoning, Translation, Classification, and Sentiment Analysis. The dataset incorporates Urdu-native Chain-of-Thought prompts, bilingual translation tasks (Urdu-English, Urdu-Urdu-English, English-Urdu-English, English-English-Urdu), and ethical considerations aligned with cultural norms.
Evaluations on Urdu-translated benchmarks, including MGSM (math reasoning), AlpacaEval (instruction-following), and Dolly General QA (open-ended questions), show that Alif-1.0-8B-Instruct consistently outperforms the base Llama model and other leading multilingual LLMs. It achieves an overall average score of 75.5 across these benchmarks, demonstrating superior reasoning, comprehension, and instruction-following capabilities in Urdu. While designed for Urdu, Alif-1.0-8B-Instruct also maintains strong performance on English benchmarks, indicating that the replay datasets effectively mitigated catastrophic forgetting, though some specialized knowledge areas saw minor degradation.
The paper also explores the impact of different quantization methods on the Alif-1.0-8B-Instruct model, using llama.cpp. Higher-bit quantization formats (6-bit and 8-bit) maintain similar perplexity levels to FP16 while significantly reducing model size, making them suitable for deployment on personal computers. Lower-bit quantization offers efficiency for resource-limited scenarios with some trade-offs in performance.
Also Read:
- GTALIGN: A Game-Theory Approach to Enhancing LLM Assistant Interactions
- Unlocking LLM Reasoning Power with Just 90 Examples: A New Data Augmentation Method
In conclusion, Alif-1.0-8B-Instruct represents a significant step forward in Urdu NLP. It addresses challenges like data scarcity and translation quality by using a high-quality multilingual synthetic dataset and a modified self-instruct approach. This work paves the way for culturally aligned, reproducible, and cost-effective Urdu NLP, fostering inclusive and trustworthy AI. For more details, you can refer to the original research paper: Alif: Advancing Urdu Large Language Models via Multilingual Synthetic Data Distillation.
