TLDR: A new tokenization method called MoVoC (Morpheme-aware Subword Vocabulary Construction) is introduced for Ge’ez script languages. It combines morpheme-based analysis with subword encoding (BPE) to better preserve morphological boundaries, which are crucial for complex, low-resource languages. While translation quality gains are modest, MoVoC significantly improves intrinsic metrics like MorphoScore and Boundary Precision, leading to more linguistically accurate and efficient tokens. The research also releases new morpheme-annotated datasets for four Ge’ez script languages.
Natural Language Processing (NLP) has made incredible strides, but it often faces significant hurdles when dealing with languages that have complex morphological structures and limited digital resources. This is particularly true for languages written in the ancient Ge’ez script, such as Amharic, Tigrinya, Ge’ez, and Tigre.
Traditional subword tokenization methods, like Byte Pair Encoding (BPE), are widely used to break down words into smaller units. While effective for many languages, these methods often struggle with morphologically rich languages. They tend to fragment words in ways that obscure their grammatical and semantic meaning, leading to less efficient and less accurate NLP models. For instance, a single verb in Amharic can change its form significantly to convey tense, aspect, person, number, and gender, making it challenging for standard tokenizers to capture these nuances.
Introducing MoVoC: A Morphology-Aware Solution
To address these challenges, researchers have developed MoVoC (Morpheme-aware Subword Vocabulary Construction). This innovative approach integrates supervised morphological analysis directly into the subword vocabulary creation process. The goal is to create a tokenizer, MoVoC-Tok, that combines the benefits of morpheme-based and BPE tokens, ensuring that the linguistic integrity of words is preserved while maintaining lexical meaning.
The MoVoC pipeline involves several key steps. First, it uses a pre-tokenization stage that includes supervised morphological analysis. For languages like Amharic and Tigrinya, existing rule-based analyzers like HornMorpho are leveraged. For Ge’ez and Tigre, where such tools are scarce, manual annotation by linguistic experts is performed to identify morpheme boundaries. This crucial step ensures that the system understands the meaningful units within words.
Next, the vocabulary construction phase combines a predefined proportion of morpheme units with BPE tokens. This hybrid strategy balances the need for linguistic accuracy with statistical efficiency. Finally, MoVoC-Tok trains a BPE tokenizer but with a critical constraint: merge operations are only permitted if they do not cross the pre-defined morpheme boundaries. This ensures that the resulting tokens are linguistically coherent.
Evaluating the Impact
The effectiveness of MoVoC-Tok was evaluated using both intrinsic and extrinsic metrics across the four Ge’ez script languages. Intrinsic evaluations, which measure how well the tokenizer preserves morpheme boundaries, showed significant improvements. MoVoC-Tok achieved substantially higher MorphScores and better Morpheme Boundary Precision compared to standard BPE, especially for the less-resourced languages like Tigre and Ge’ez. This indicates that the hybrid approach successfully instills morpheme awareness into the tokenization process.
In terms of extrinsic evaluation, specifically machine translation (MT) performance from English to the target languages, MoVoC-Tok consistently outperformed baseline tokenizers like BPE and WordPiece. While the gains in automatic translation quality metrics (BLEU and chrF++) were modest, they were consistent across all tasks.
A qualitative analysis further highlighted MoVoC’s practical benefits. It demonstrated how MoVoC-Tok preserves morphological integrity, preventing the fragmentation of words into arbitrary subword units. For example, the Tigrinya word for “do not do it” was segmented more meaningfully by MoVoC-Tok, preserving its negation prefix, verb root, and suffix. This leads to improved alignment in translation and better handling of rare or derived word forms, allowing models to generalize knowledge across related words.
Also Read:
- Bridging the Language Gap: Enhancing Fairness in Multilingual Search Systems
- COMPACT: A Dual Pruning Strategy for Efficient and Deployable Large Language Models
Future Directions and Resources
This research not only proposes a novel tokenization method but also contributes valuable resources to the NLP community. The authors are releasing manually annotated morpheme datasets for Amharic, Tigrinya, Ge’ez, and Tigre, along with morpheme-aware vocabularies for two of them. These resources are crucial for supporting further research in low-resource, morphologically rich languages.
While MoVoC-Tok represents a significant step forward, the researchers acknowledge limitations, including modest gains in overall translation quality and the generalizability of the method due to the small set of languages studied. Nevertheless, this work underscores the importance of integrating linguistic knowledge into tokenization for languages where morphology plays a central role. For more details, you can refer to the full research paper here.
