TLDR: A new research paper introduces Greek Embedding Models (GEMs), a family of advanced transformer models designed to overcome limitations in Modern Greek Natural Language Processing (NLP). By curating extensive, high-quality Greek corpora and employing modern architectures like ELECTRA and ConvBERT, the GEMs achieve significant performance improvements over existing models in tasks such as named entity recognition and legal topic classification, particularly within the complex legal domain. The research also explores bilingual Greek-English models and highlights the importance of data quality and architectural diversity for moderately-resourced languages.
A new research paper titled “Forging GEMs: Advancing Greek NLP Through Quality-Based Corpus Curation and Specialized Pre-training” introduces a significant leap forward in Natural Language Processing (NLP) for Modern Greek. Authored by Alexandra Apostolopoulou, Konstantinos Kanaris, Athanasios Koursaris, George Domalis, Dimitris Tsakalidis, and Ioannis E. Livieris, this work addresses long-standing challenges in processing a morphologically rich, moderately-resourced language like Greek, especially in specialized fields such as law.
Historically, the development of Greek NLP has faced hurdles including a fragmented research landscape, a lack of diverse architectural approaches, and a reliance on models with limited context-length capabilities. This has been particularly problematic in domains like law, where lengthy and complex documents require models capable of understanding extensive textual information. Existing models often used older transformer architectures with a restrictive 512-token window, which proved insufficient for comprehensive analysis.
To tackle these issues, the researchers propose Greek Embedding Models (GEMs), a novel family of transformer models built on a foundation of extensive, quality-driven data curation. The project involved meticulously constructing several large-scale Greek corpora, which included both general-domain texts and specialized legal sources. A rigorous, quality-based filtering and preprocessing methodology was applied to create high-value training datasets, ensuring the models learned from clean and relevant information.
Architectural Innovations and Diverse Models
The paper details the pre-training and systematic evaluation of a diverse suite of modern transformer architectures that had not been previously applied to the Greek language. These include advanced models such as ELECTRA, ConvBERT, and ModernBERT, alongside established ones like RoBERTa and Longformer. This broad exploration of architectures aims to counter the lack of diversity in the Greek NLP space.
A key innovation is the systematic training and evaluation of these Greek models with an extended 1024-token context window, a substantial improvement over previous limitations. Furthermore, the research introduces the first bilingual Greek-English Embedding Models (GEM*) specifically designed for the legal domain, opening avenues for cross-lingual understanding.
Key Contributions and Performance
The main contributions of this work are multifaceted:
- The development of the GEMs family, encompassing both established and advanced architectures, all pre-trained from scratch on curated corpora.
- The construction and release of large-scale, meticulously curated corpora, addressing data scarcity for the Greek language.
- Investigation of data-centric strategies, demonstrating that targeted repetition of high-quality legal sub-corpora is effective for domain adaptation.
- The introduction of bilingual Greek-English Embedding Models (GEM*) tailored for the legal domain.
- A comprehensive comparative analysis on diverse downstream tasks, including Named Entity Recognition (NER), Multi-Class Legal Topic Classification (MCLTC), and Natural Language Inference (NLI).
Extensive experiments on these downstream tasks demonstrated that the new class of GEMs significantly outperforms existing baselines. Notably, the GEM-RoBERTa and GEM-ConvBERT models showed superior performance across various benchmarks. GEM-ELECTRA and GEM-ModernBERT also reported competitive results, validating the effectiveness of these modern architectural innovations for Greek language processing. Interestingly, the study found that the optimal training corpus varied by task: domain-specific legal corpora were best for specialized tasks like NER and MCLTC, while a general-domain corpus yielded better results for broad language understanding tasks like NLI.
The research also highlighted the importance of tokenization strategy for morphologically rich languages, with the WordPiece tokenizer trained on the Legal Corpus achieving the best balance between low fragmentation and high compression efficiency for Greek legal text.
Also Read:
- A New Standard for Legal AI: Unveiling the Massive Legal Embedding Benchmark
- Unlocking Ancient Tongues: AI’s Role in Discovering Latin in Historical Texts
Future Directions
While the extended 1024-token context window is an improvement, the authors acknowledge that it remains insufficient for processing entire legal documents, which can span thousands of tokens. Future work will focus on applying these models to additional tasks such as summarization, question answering, and information retrieval, and extending the approach to other specialized domains like biomedicine and finance. The promising bilingual results also suggest exploring multilingual extensions to incorporate other European languages with comparable legal systems. For more details, you can read the full paper here.
