TLDR: This research addresses the challenge of sentiment analysis for Persian, a low-resource language, by combining cross-lingual few-shot learning with incremental adaptation. It leverages pre-trained multilingual models (XLM-RoBERTa, mDeBERTa, DistilBERT) and fine-tunes them on small, diverse Persian datasets. The study demonstrates that mDeBERTa and XLM-RoBERTa, particularly when using regularization techniques like knowledge distillation and rehearsal, achieve high accuracy (up to 96%), proving that effective sentiment analysis is possible in data-scarce environments and can be extended to other low-resource languages.
Sentiment analysis, the process of automatically detecting and classifying emotions in text, has seen significant progress for widely spoken languages like English. However, many languages, including Persian, face a major challenge: a severe lack of labeled data. This scarcity makes it difficult to train effective sentiment analysis models, highlighting the need for methods that can perform well with minimal supervision.
A recent research paper, titled “Cross-lingual Few-shot Learning for Persian Sentiment Analysis with Incremental Adaptation,” tackles this very issue. Authored by Farideh Majidi and Ziaeddin Beheshtifard from the Islamic Azad University, South Tehran Branch, this study explores a novel approach to enable sentiment analysis in Persian using limited data by leveraging knowledge from high-resource languages.
The Core Problem: Data Scarcity
Traditional machine learning models require vast amounts of labeled data to learn effectively. For languages like Persian, gathering and annotating such large datasets is time-consuming and expensive. This research aims to overcome this by employing strategies that allow models to learn from very few examples (few-shot learning) and adapt gradually over time (incremental learning).
The Solution: Multilingual Models and Adaptive Learning
The researchers utilized three pre-trained multilingual models: XLM-RoBERTa, mDeBERTa, and DistilBERT. These models are initially trained on massive text corpora from multiple languages, allowing them to transfer learned linguistic patterns from resource-rich languages to low-resource ones like Persian. The key idea is to fine-tune these powerful models on only a small amount of Persian data.
To ensure the models could handle diverse real-world Persian text, they were fine-tuned using small samples from various sources, including social media platforms like X (formerly Twitter) and Instagram, and e-commerce/review sites like Digikala, Snappfood, and Taaghche. This variety helps the models learn from a broad range of contexts and writing styles.
Incremental Adaptation and Forgetting Prevention
A crucial aspect of this research is incremental learning. This involves introducing data from different domains sequentially, allowing the model to gradually build knowledge and adapt to new characteristics of the Persian language. To prevent “catastrophic forgetting”—where models forget previously learned information when exposed to new data—several regularization techniques were employed: Elastic Weight Consolidation (EWC), knowledge distillation, and rehearsal. These methods help the model retain essential knowledge while learning new information.
Also Read:
- Bridging Language Gaps: A New Approach to Translating Informal and Toxic Content
- Enhancing LLM Safety Across Languages with SEALGUARD
Key Findings and Performance
The experimental results were highly promising. The mDeBERTa and XLM-RoBERTa models demonstrated strong performance, achieving impressive accuracy rates of up to 96% on Persian sentiment analysis. This highlights the effectiveness of combining few-shot learning and incremental learning with multilingual pre-trained models.
Specifically, Knowledge Distillation showed the best performance in extreme few-shot (1-shot) settings across all models, particularly for XLM-RoBERTa and mDeBERTa. Rehearsal consistently performed well across various data sizes and often matched or even surpassed scenarios where no incremental learning was applied, especially with mDeBERTa. DistilBERT, being a smaller model, generally underperformed compared to the other two, indicating that multilingual pre-training is crucial in low-shot scenarios.
The study concludes that incremental learning methods, especially when combined with regularization techniques like Rehearsal and Knowledge Distillation, significantly improve model performance and help retain past knowledge while adapting to new data. This approach offers a promising pathway for developing effective sentiment analysis systems for Persian and can be extended to other low-resource languages facing similar challenges.
For a deeper dive into the methodology and detailed results, you can read the full research paper here.
