TLDR: CultranAI participated in the PalmX 2025 shared task, focusing on data augmentation and LoRA fine-tuning of LLMs for Arabic cultural knowledge. They created a new dataset of over 22K culturally grounded multiple-choice questions and benchmarked several LLMs, finding Fanar-1-9B-Instruct to be the best. Their system achieved 70.50% accuracy on the blind test set, demonstrating the effectiveness of their data augmentation strategy.
Large Language Models (LLMs) are increasingly used for everyday queries, many of which involve local culture, traditions, and languages. However, these models often struggle with culturally specific information, especially in diverse linguistic environments like Arabic, which has many dialects. Addressing this gap is crucial for developing LLMs that are truly aligned with human understanding and interaction.
The research paper, titled “CultranAI at PalmX 2025: Data Augmentation for Cultural Knowledge Representation,” details the participation of the CultranAI system in the PalmX cultural evaluation shared task. This task specifically aims to benchmark Arabic language models on their ability to answer culturally grounded multiple-choice questions in Modern Standard Arabic (MSA). The questions cover a wide range of topics, including history, customs, geography, literature, and food, reflecting general cultural literacy in Arab countries.
The CultranAI team, comprising Hunzalah Hassan Bhatti, Youssef Ahmed, Md Arid Hasan, and Firoj Alam, focused on enhancing LLMs’ understanding of Arabic cultural knowledge. Their approach centered on two main strategies: data augmentation and LoRA (Low-Rank Adaptation) fine-tuning of LLMs.
Building a Richer Dataset
A significant part of their work involved expanding the training data. Beyond the provided PalmX dataset, they incorporated the Palm dataset, a community-curated resource spanning MSA and various Arabic dialects. To make this data compatible with the PalmX format, they converted its instruction-style question-answer pairs into multiple-choice questions using GPT-4.1, generating plausible incorrect answer options (distractors).
Furthermore, they curated a new, extensive dataset called PalmX-ext, consisting of over 22,000 culturally grounded multiple-choice questions. This was achieved by leveraging the NativQA framework in combination with GPT-4.1. The NativQA framework helped curate location-specific QA pairs, ensuring cultural and regional alignment. All answers were filtered for factual quality using NativQA’s Domain Reliability Check and further refined by GPT-4.1 to remove irrelevant or incorrect information.
Benchmarking and Fine-tuning LLMs
The researchers evaluated several open and closed LLMs in a zero-shot setting to identify the best-performing model for the task. This initial comparison included models like Qwen2.5-7B-Instruct, Jais-13B-Chat, Llama-3.1-8B-Instruct, and Fanar-7B-Instruct. The Fanar-7B-Instruct model emerged as the top performer on the PalmX development set, making it the chosen model for further fine-tuning.
For fine-tuning, they experimented with two PEFT (Parameter-Efficient Fine-Tuning) methods: LoRA and QLoRA. LoRA trains a small set of adapter layers while keeping the main model frozen, while QLoRA combines 4-bit quantization with LoRA to reduce memory usage. Their experiments showed that LoRA achieved a slightly better performance than QLoRA, indicating that full-precision adapters were marginally more effective.
Impact of Data Augmentation
A key finding of the study was the substantial positive impact of their data augmentation strategy. Training the Fanar-7B-Instruct model on the combined augmented dataset (PalmX Train, Palm, and PalmX-ext) led to significant performance gains compared to training only on the original PalmX Train data. This demonstrated that a richer, more diverse dataset greatly improves the model’s ability to generalize and understand cultural nuances.
Also Read:
- Unpacking AI Performance in Moroccan Legal Question Answering
- Assessing AI’s Medical Acumen in Arabic Healthcare
Results and Future Directions
The CultranAI system achieved an accuracy of 84.1% on the PalmX development set. On the blind test set, their submitted system ranked 5th, with an accuracy of 70.50%. These results highlight the effectiveness of their data augmentation and fine-tuning approach in enhancing cultural knowledge representation in LLMs for Arabic.
The authors acknowledge certain limitations, such as issues with distractor quality and token limits in some converted questions, which they plan to address in future work. They also aim to further refine data augmentation pipelines and explore model generalizability. For more details, you can read the full research paper here.
