Forging GEMs: Advancing Greek NLP through Quality-Based Corpus Curation and Specialized Pre-training

2025-10-22

AI Summary

Modern Greek—morphologically rich and moderately resourced—faces three key bottlenecks in high-stakes NLP applications (e.g., legal domains): fragmented research, outdated architectures (largely relying on early Transformer variants), and limited context length (typically ≤512 tokens). Method: We construct a high-quality bilingual (Greek–English) legal corpus and introduce GEM, the first Greek-specific pre-trained model family tailored for the legal domain. GEM innovatively integrates state-of-the-art architectures—including ELECTRA, ConvBERT, and ModernBERT—into Greek language modeling, leveraging quality-driven corpus filtering and large-scale preprocessing. Contribution/Results: Experiments demonstrate that GEM-RoBERTa and GEM-ConvBERT significantly outperform existing baselines across multiple legal downstream tasks, empirically validating the effectiveness of the “high-quality domain corpus + modern architecture” paradigm for under-resourced, morphologically complex languages.

Technology Category

• Natural Language Processing: Lexical Semantics and Morphology
• Machine Learning: Large Multimodal Models (LMMs)
• Application Domains: Humanities & Computational Social Science

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The advancement of natural language processing for morphologically rich, moderately-resourced languages like Modern Greek is often hindered by a fragmented research landscape, a lack of architectural diversity and reliance on limited context-length models. This is particularly true in specialized, high-value domains such as law, where existing models are frequently confined to early transformer architectures with a restrictive 512-token window, insufficient for analyzing long legal documents. To address these challenges, this paper presents Greek Embedding Models, a new family of transformer models for Greek language built upon a foundation of extensive, quality-driven data curation. We detail the construction of several large-scale Greek corpora, emphasizing a rigorous, quality-based filtering and preprocessing methodology to create high-value training datasets from both general-domain and specialized legal sources. On this carefully curated foundation, we pre-train and systematically evaluate a diverse suite of modern architectures, which has not previously applied to Greek language, such as ELECTRA, ConvBERT and ModernBERT. Furthermore, we propose the first bilingual Greek-English Embedding Models tailored for the legal domain. The extensive experiments on downstream tasks demonstrate that the new class of models establish the effectiveness of the proposed approach, highlighting that the GEM-RoBERTa and GEM-ConvBERT models significantly outperform existing baselines.

Problem

• Advancing Greek NLP with quality-curated corpora and diverse architectures
• Addressing limitations in legal domain models with extended context windows
• Developing bilingual Greek-English models for specialized legal applications

Innovation

• Quality-based corpus curation methodology for Greek
• Pre-training diverse modern architectures like ELECTRA
• First bilingual Greek-English legal domain models