TLDR: CETVEL is a new, comprehensive benchmark for evaluating Turkish LLMs, featuring 23 diverse tasks across understanding and generation, with a strong focus on cultural relevance. It reveals that while general-purpose LLMs often outperform Turkish-centric ones, specific models like Cere-Llama-3-8B excel in culturally-grounded tasks. Grammatical error correction, machine translation, and extractive QA are identified as the most discriminative tasks, providing valuable insights for advancing Turkish NLP.
Large Language Models (LLMs) have shown impressive capabilities in English, but their performance in other languages, especially those with rich linguistic and cultural nuances like Turkish, often remains underexplored. Addressing this critical gap, researchers have introduced CETVEL, a new and comprehensive benchmark designed specifically to evaluate LLMs for the Turkish language.
Existing Turkish benchmarks frequently fall short in either task diversity or culturally relevant content, or both. CETVEL aims to overcome these limitations by offering a broad spectrum of both discriminative and generative tasks. This ensures that the evaluation content truly reflects the unique linguistic and cultural richness of Turkish.
What Does CETVEL Cover?
The benchmark encompasses 23 distinct tasks, organized into seven main categories. These categories include Text Classification (TC), Multiple Choice Question Answering (MCQA), Extractive Question Answering (QA), Grammatical Correction (GC), Machine Translation (MT), Summarization (SUM), and Natural Language Inference (NLI). The tasks range from fundamental language understanding to complex generation, featuring challenges like grammatical error correction, machine translation, and question answering deeply rooted in Turkish history and idiomatic expressions. For instance, it includes tasks like identifying the meaning of Turkish proverbs, solving riddles, and disambiguating words based on circumflex diacritics, which are crucial for Turkish linguistic specificity.
Evaluating LLMs for Turkish
The researchers evaluated 33 open-weight LLMs, with parameter sizes up to 70 billion, covering various model families and instruction paradigms. This extensive evaluation included general-purpose models like Llama 3 and Mistral, multilingual LLMs such as Aya and Qwen2.5, and Turkish-centric models like Kanarya, Turna, Commencis-LLM-7B, Trendyol-LLM-7B, and Cere-Llama-3-8B.
The findings from CETVEL reveal some interesting trends. Generally, Turkish-centric instruction-tuned models tend to underperform compared to multilingual or general-purpose models, even though they are specifically designed for Turkish. This suggests a need for improved instruction-tuning, continued pretraining, and more rigorous validation strategies for these specialized models.
However, there are exceptions. Cere-Llama-3-8B, a Turkish-centric model, demonstrated exceptional performance in grammatical error correction and extractive question answering related to Turkish and Islamic history. It even surpassed the larger 70-billion parameter Llama-3.3-70B-Instruct in these specific areas, highlighting the benefits of targeted tuning for culturally grounded datasets. Conversely, Cere-Llama-3-8B showed weaker performance in machine translation and knowledge-intensive tasks, likely due to less exposure to English and general-domain fine-tuning.
Also Read:
- Unpacking AI’s Grasp of Human Reasoning Styles in Social Games
- Addressing Age Bias in Medical AI: Introducing the PediatricsMQA Benchmark
Most Informative Tasks
To understand which tasks are most effective in differentiating model capabilities, a Gini coefficient-based analysis was performed. This analysis indicated that grammatical error correction, machine translation, and extractive question answering are particularly discriminative. These tasks consistently showed wide performance gaps across different models, making them highly valuable for benchmarking LLMs in Turkish. In contrast, tasks like natural language inference and text classification were found to be less effective in distinguishing model strengths.
CETVEL stands as a significant step forward, offering a comprehensive and culturally grounded evaluation suite that will be instrumental in advancing the development and assessment of LLMs for the Turkish language. For more details, you can refer to the original research paper: CETVEL: A Unified Benchmark for Turkish LLMs.
