TLDR: This research explores the effectiveness of large language models (LLMs) in identifying hope, hate speech, offensive language, and emotional expressions in Arabic textual speech and multi-modal memes. Evaluating base, fine-tuned, and embedding models on datasets from the ArabicNLP MAHED 2025 challenge, the study found that fine-tuned LLMs like GPT-4o-mini and Gemini Flash 2.5 achieved superior performance, offering improved tools for nuanced Arabic content moderation.
The digital landscape, particularly social media, has become a primary channel for expression in the Arabic-speaking world. While this fosters connection and information sharing, it also presents a significant challenge: the rapid spread of offensive language, hate speech, and emotionally charged content, often embedded within textual posts and multi-modal memes. This growing concern highlights an urgent need for advanced content moderation systems capable of accurately analyzing such complex Arabic content.
A recent research paper, “Detecting Hope, Hate, and Emotion in Arabic Textual Speech and Multi-modal Memes Using Large Language Models,” by Nouar AlDahoula and Yasir Zakia, delves into this critical area. The study investigates the potential of large language models (LLMs) to effectively identify hope, hate speech, offensive language, and various emotional expressions within Arabic digital content. This research is particularly vital given the unique challenges of Arabic content moderation, including its rich dialect diversity and the scarcity of specialized tools and training data.
Addressing the Nuances of Arabic Online Content
The paper focuses on three distinct but related tasks, utilizing datasets from the ArabicNLP MAHED 2025 challenge:
- Hope and Hate Detection in Arabic Textual Speech: This task involved classifying text into categories of hope, hate, or not_applicable. The researchers fine-tuned LLMs like GPT-4o-mini and Gemini Flash 2.5, and also explored embedding models combined with traditional classifiers like Support Vector Machines (SVM). An ensemble method, which combined the predictions of multiple models, significantly boosted performance.
- Multi-task Detection of Emotional Expressions, Offensive Language, and Hate Speech in Arabic Text: Here, the goal was to identify twelve different emotions (e.g., anger, joy, sadness), detect offensive language (yes/no), and further classify offensive content as hate or not_hate. Fine-tuned GPT-4o-mini models were employed for these sub-tasks, demonstrating strong capabilities in understanding the emotional landscape of Arabic text.
- Multi-modal Detection of Arabic Hateful Memes: Perhaps the most complex task, this involved analyzing both the visual and textual components of Arabic memes to determine if they contained hate speech. The study evaluated a range of models, including base LLMs, fine-tuned LLMs (Gemini Flash 2.5, Llama 3.2-11B, PaliGemma2), and multi-modal embedding models with SVM or Deep Neural Networks (DNN). Existing safety classifiers like Llama Guard 4 and OpenAI’s content moderator were also assessed, revealing their limitations in this specific context.
Key Findings and Model Performance
The research yielded promising results, underscoring the power of LLMs in this domain:
- For hope/hate detection in textual speech, an ensemble approach combining fine-tuned GPT-4o-mini, Gemini Flash 2.5, and Google text embedding + SVM achieved the highest Macro F1 score of 72.1%.
- In the multi-task detection of emotions, offensive language, and hate speech, fine-tuned GPT-4o-mini models demonstrated superior performance, achieving a Macro F1 score of 57.8% across the sub-tasks.
- Crucially, for the multi-modal detection of hateful memes, fine-tuned Gemini Flash 2.5 emerged as the top performer with an impressive Macro F1 score of 79.6%. Fine-tuned Llama 3.2-11B also showed strong results, ranking second. This highlights that fine-tuning LLMs with specific Arabic meme data significantly enhances their ability to detect hate speech in this complex multi-modal format.
- Interestingly, while base LLMs like GPT-4o-mini showed good performance, existing general-purpose safety classifiers like Llama Guard 4 and OpenAI’s content moderator exhibited lower recall and F1-scores, indicating their limitations when dealing with the specific nuances of Arabic hateful memes.
The study also explored the effectiveness of multi-modal embedding models, finding that combining image and text embeddings with SVM or DNN classifiers was effective, with the average embedding vector slightly outperforming image-only embeddings. This suggests that Google’s embedding model might already process text within meme images effectively.
Also Read:
- MM-ORIENT: A New AI Framework for Deeper Multimodal Content Understanding
- Unpacking AI Performance in Moroccan Legal Question Answering
Implications for Content Moderation
The findings of this paper are significant for the development of more accurate and efficient Arabic content moderation systems. By demonstrating the superior capacity of fine-tuned LLMs, particularly GPT-4o-mini for textual speech and Gemini Flash 2.5 for multi-modal memes, the research provides a pathway for better understanding and managing the vast and diverse content generated in Arabic online spaces. These solutions offer a more nuanced approach, which is essential for creating safer, more respectful, and law-abiding online environments across Arabic-speaking regions.
Despite the advancements, the authors acknowledge limitations, including the subjective nature of content annotations and the ongoing challenge of addressing the wide array of Arabic dialects. Future work will likely focus on overcoming these hurdles to further refine AI-powered content moderation. For a deeper dive into the methodologies and detailed results, you can access the full research paper here.
