The Degree of Language Diacriticity and Its Effect on Tasks

2026-03-29

AI Summary

This study addresses the lack of cross-linguistic, data-driven frameworks for quantifying the dependence of diacritics on writing systems and their impact on natural language processing (NLP). The authors propose the first corpus-based information-theoretic framework that defines diacritic complexity along three dimensions: frequency, ambiguity, and structural diversity. They evaluate this framework through diacritic restoration experiments using BERT and RNN models across 24 corpora spanning 15 languages. Results demonstrate that higher diacritic complexity correlates with lower restoration accuracy. Notably, in languages with multiple diacritics, structural complexity proves a stronger predictor of model performance than frequency-based measures, underscoring its critical role in NLP tasks.

Abstract

Diacritics are orthographic marks that clarify pronunciation, distinguish similar words, or alter meaning. They play a central role in many writing systems, yet their impact on language technology has not been systematically quantified across scripts. While prior work has examined diacritics in individual languages, there's no cross-linguistic, data-driven framework for measuring the degree to which writing systems rely on them and how this affects downstream tasks. We propose a data-driven framework for quantifying diacritic complexity using corpus-level, information-theoretic metrics that capture the frequency, ambiguity, and structural diversity of character-diacritic combinations. We compute these metrics over 24 corpora in 15 languages, spanning both single- and multi-diacritic scripts. We then examine how diacritic complexity correlates with performance on the task of diacritics restoration, evaluating BERT- and RNN-based models. We find that across languages, higher diacritic complexity is strongly associated with lower restoration accuracy. In single-diacritic scripts, where character-diacritic combinations are more predictable, frequency-based and structural measures largely align. In multi-diacritic scripts, however, structural complexity exhibits the strongest association with performance, surpassing frequency-based measures. These findings show that measurable properties of diacritic usage influence the performance of diacritic restoration models, demonstrating that orthographic complexity is not only descriptive but functionally relevant for modeling.

Problem

• diacritics
• language technology
• orthographic complexity
• cross-linguistic
• diacritic restoration

Innovation

• diacritic complexity
• information-theoretic metrics
• cross-linguistic analysis
• diacritics restoration
• orthographic structure