Expanding the Lexicon of Ge'ez Based African Languages: A Comparative Study of Amharic and Tigrinya

2026-07-16

AI Summary

This work addresses the poor performance of multilingual pretrained models on low-resource, non-Latin scripts such as Ge'ez due to high out-of-vocabulary rates and subword fragmentation. To mitigate this, the authors propose a language-specific SentencePiece tokenizer for Amharic and Tigrinya, which extends the XLM-R vocabulary and initializes new tokens via subword embedding averaging. Coupled with a two-stage continual pretraining strategy, this approach significantly enhances the model’s comprehension of Ge'ez-script languages and generalizes effectively to 17 related low-resource African languages. Experimental results demonstrate substantial improvements: the method achieves 87.0 EM and 90.0 F1 on question answering, 80.0% accuracy in sentiment analysis, and boosts out-of-vocabulary entity recognition accuracy from 81.4% to 94.3%, consistently outperforming XLM-R and Glot500 baselines.

Abstract

Multilingual pre-trained language models (PLMs) exhibit degraded performance on low-resource, non-Latin-script languages, driven by high out-of-vocabulary (OOV) rates and excessive subword fragmentation that result from Latin-script-centric tokenizer training. We introduce VEXMLM, a vocabulary-extended variant of XLM-R targeting the two highest-resource Ge'ez-script languages, Amharic and Tigrinya, and further evaluated on 17 additional low-resource African languages (19 total). We train a language-specific SentencePiece tokenizer on curated Amharic and Tigrinya monolingual corpora, extend XLM-R's vocabulary with 30,000 Ge'ez-script subwords derived from this tokenizer, and initialize their embeddings by averaging the embeddings of their constituent subwords under XLM-R's original tokenizer. VEXMLM is trained in two stages: (1) continued masked language modeling over the extended vocabulary on the curated corpora, and (2) supervised fine-tuning on question answering (QA), named entity recognition (NER), and sentiment analysis (SA). On Amharic/Tigrinya QA, VEXMLM achieves 87.0 EM /90.0 F1, versus 66.0 EM/78.0 F1 for XLM-R and 74.0 EM/ 78.0 F1 for Glot500. On SA, VEXMLM reaches 80.0\% accuracy versus 77.0\% (XLM-R) and 46.0\% (Glot500). On NER, VEXMLM raises OOV-token entity accuracy from 81.4\% to 94.3\%, averaged over 11 of the 19 evaluated languages for which OOV analysis was possible. Our contributions are: (i) a vocabulary-extension and embedding-initialization procedure tailored to Ge'ez script; (ii) a two-stage training strategy under which vocabulary and continued-pretraining gains on Amharic/Tigrinya transfer to 17 typologically related, unaugmented African languages; and (iii) an evaluation spanning both intrinsic tokenization metrics (vocabulary coverage, fertility, OOV rate) and extrinsic task performance across all 19 languages.

Problem

• low-resource languages
• Ge'ez script
• out-of-vocabulary
• subword fragmentation
• multilingual pre-trained language models

Innovation

• vocabulary extension
• Ge'ez script
• low-resource languages
• multilingual pre-trained language models
• tokenization