TLDR: A new research paper comprehensively benchmarks eleven open-source Large Language Models (LLMs) for Persian Natural Language Processing (NLP) tasks under zero-shot and few-shot learning conditions. The study found that Gemma2 consistently outperformed other models across various tasks, including sentiment analysis, named entity recognition, reading comprehension, and question answering. While few-shot learning generally improved performance, token-level tasks like Named Entity Recognition remained challenging. The research highlights the persistent performance gap between Persian and high-resource languages, emphasizing the need for more targeted development and evaluation of LLMs for low-resource linguistic communities.
Large Language Models (LLMs) have transformed how we interact with technology, showing incredible abilities in understanding and generating human language. While these models excel in widely spoken languages like English, their performance in languages with fewer digital resources, such as Persian (Farsi), often lags. This disparity highlights a critical need for dedicated research and evaluation in these low-resource linguistic contexts.
A recent study, titled BENCHMARKING OPEN-SOURCE LARGE LANGUAGE MODELS FOR PERSIAN IN ZERO-SHOT AND FEW-SHOT LEARNING, addresses this gap by providing a comprehensive benchmark of several open-source LLMs specifically for Persian Natural Language Processing (NLP) tasks. The research explores how these models perform under both zero-shot and few-shot learning conditions, which are crucial for languages where extensive training data is scarce.
Understanding the Challenge of Persian
Persian presents unique challenges for NLP due to its distinct writing system, complex word structures (morphology), and sentence patterns (syntax) that differ significantly from many Indo-European languages. Compounding this is the limited availability of high-quality Persian datasets, annotated text, and specialized NLP tools compared to those available for languages like English. This makes it difficult to train and evaluate LLMs effectively for Persian-speaking communities.
How the Models Were Evaluated
The researchers evaluated eleven prominent open-source LLMs, including Gemma2, GLM4, LLaMA3.1, LLaMA3.2, Qwen2, Qwen2.5, Mistral, Marcoo-O1, Aya-Expanse, Falcon3, and Tulu3. These models represent a diverse range of architectures and training approaches. The evaluation focused on a variety of Persian NLP tasks:
- Sentiment analysis (determining emotion in text)
- Named Entity Recognition (identifying names of people, places, organizations)
- Reading comprehension
- Question answering
- Machine translation
- Text summarization
- Logical, commonsense, and mathematical reasoning
These tasks utilized established Persian datasets like ParsiNLU, ArmanEmo, ArmanNER, and a Persian adaptation of the Massive Multitask Language Understanding (MMLU) benchmark. The evaluation was conducted in two main settings:
- Zero-shot learning: Models performed tasks without any specific training examples, relying solely on their pre-trained knowledge and general instructions.
- Few-shot learning: Models were given a minimal number of illustrative examples (typically 5) for each task to see how quickly they could adapt.
Key Findings: Gemma2 Leads the Pack
The study revealed that Gemma2 consistently outperformed all other models across nearly all tasks in both zero-shot and few-shot learning scenarios. It achieved the highest average scores, demonstrating superior generalization capabilities and adaptability to Persian language nuances. This suggests that Gemma2’s design and multilingual pretraining strategies are particularly effective for languages like Persian.
Generally, few-shot learning led to better results for most models, with significant improvements observed in complex reasoning and comprehension tasks. This highlights the value of providing even a small number of examples to help models adapt to specific Persian linguistic phenomena.
However, the study also identified persistent challenges. Tasks requiring token-level understanding, such as Named Entity Recognition, proved difficult for most models, showing only modest improvements even with few-shot prompting. This indicates a need for specialized architectural adaptations for morphologically rich languages like Persian.
Also Read:
- Large Language Models Bring Context to Text Preprocessing
- NARRABENCH: A New Framework to Assess AI’s Grasp of Stories
Challenges and Future Directions
Despite the progress, a notable performance gap persists between Persian and comparable English benchmarks, with Persian scores averaging 18.7% lower. Error analysis showed models struggled with semantic ambiguities, nuanced sentiments, precise entity boundary detection, and Persian-specific idiomatic expressions. These findings underscore the ongoing challenges of linguistic underrepresentation in large-scale training data.
The researchers suggest several future directions, including evaluating more Persian-specific LLMs (like Maral and PersianMind), expanding to more complex tasks, exploring advanced prompting techniques, and developing domain-specific Persian benchmarks. The goal is to foster transparency, reproducibility, and collaborative progress in Persian NLP, ultimately working towards truly multilingual systems that serve diverse user populations equitably.
