TLDR: This paper investigates how tokenization and information representation biases in multilingual LLMs affect their performance on dialectal NLP tasks. It finds that encoder models generally outperform decoder models. Tokenization Parity (TP) predicts performance for syntactic tasks, while Information Parity (IP) predicts semantic tasks. Surprisingly, higher TP can aid dialect identification by highlighting surface differences, while higher IP can hinder it by making dialects too semantically similar. The study reveals that LLM language support claims often mask deeper token-level mismatches, especially for non-Latin scripts.
Large Language Models (LLMs) have become indispensable for various Natural Language Processing (NLP) tasks, but their performance often falters when dealing with the subtle yet significant variations found in dialects. This phenomenon, known as the “dialect gap,” highlights a critical challenge: how well do these powerful models truly understand and represent diverse linguistic forms?
A recent research paper, “Tokenization and Representation Biases in Multilingual Models on Dialectal NLP Tasks”, delves into this issue by examining two key factors: Tokenization Parity (TP) and Information Parity (IP). Authored by Vani Kanjirangat, Tanja Samardži´c, Ljiljana Dolamic, and Fabio Rinaldi, the study investigates how representational biases in pre-trained multilingual models correlate with their performance on dialectal tasks.
Understanding Tokenization and Information Parity
Tokenization Parity (TP) assesses how consistently a tokenizer processes parallel sentences across different languages. Ideally, a tokenizer should produce similar tokenized lengths for the same content, indicating good vocabulary coverage for that language. A TP value close to 1 is desirable; values significantly greater than 1 suggest that the tokenizer is breaking text into many smaller fragments, which can lead to inefficient representation and potentially hinder performance.
Information Parity (IP), on the other hand, measures an LLM’s efficiency in compressing text in one language compared to a reference language (like English). A higher IP indicates that the model is more efficient at representing the information in that language, which should intuitively lead to better performance.
Key Findings Across Dialectal Tasks
The researchers evaluated various state-of-the-art decoder-only LLMs (such as Phi-3.5 and Llama 3.2) and encoder-based models (like mBERT) across three distinct dialectal NLP tasks: dialect classification, topic classification, and extractive question answering. They also considered the impact of script type (Latin vs. non-Latin) and resource availability (high vs. low).
One striking observation was that encoder-based models consistently outperformed decoder-only LLMs across all evaluated dialectal tasks. This suggests that for the specific challenges posed by dialects, the architecture and training methodologies of encoder models might offer an advantage.
The study revealed that TP is more sensitive to the script type. Latin-script languages generally exhibited more stable TP values closer to 1, indicating better tokenization efficiency. Non-Latin languages, however, showed considerable variability, especially in low-resource settings. IP, conversely, was influenced by both script and resource availability, with high-resource Latin-script languages typically showing higher IP.
Correlations with Performance
The relationship between these parity metrics and model performance was nuanced and task-dependent:
- Extractive Question Answering (EQA): For tasks requiring precise span extraction, TP was a strong predictor. Higher TP (more fragmentation) generally correlated with worse performance, as excessive tokenization can complicate the model’s ability to identify correct answer spans.
- Topic Classification (TC): This semantic task showed a stronger correlation with IP. Higher IP, indicating more efficient information compression, was associated with improved performance, as understanding the topic requires a deeper semantic grasp.
- Dialect Identification (DI): Surprisingly, for dialect identification, a positive correlation was found between TP and performance, while higher IP correlated with worse performance. The researchers theorize that for distinguishing dialects, which often differ at surface levels (spelling, morphology), increased fragmentation (higher TP) might inadvertently help models pick up on subtle, token-level cues. Conversely, if an LLM achieves high IP, it might be compressing dialects too similarly at a semantic level, making it harder to differentiate them based on their unique surface characteristics.
Challenges with Tokenizers
The paper also highlighted specific issues, such as the Llama-3.2 tokenizer’s behavior with non-Latin scripts. It was observed to produce misaligned tokens, often misinterpreting non-Latin characters as Latin-1 due to byte-level fallbacks. This can lead to longer token sequences and degrade performance for these languages, raising questions about the model’s ability to capture semantic meaning and linguistic nuances effectively.
Also Read:
- Advancing English–Tigrinya Machine Translation with Custom Tokenizers and Refined Evaluation
- Bridging Cultural Divides: How Small Data Sets Can Adapt LLMs to Diverse Global Contexts
Conclusion
This research underscores that the “language support” claims of many LLMs might not fully account for deeper mismatches at the script or token level, especially when dealing with dialectal variations. The findings emphasize the critical role of tokenization and representation biases in multilingual NLP. Moving forward, the development of language-aware, adaptive tokenizers that can flexibly operate across different granularities will be crucial for building more equitable and effective LLMs for diverse linguistic landscapes.
