Equity with Efficiency: An Empirical Study of Tokenizers for Multilingual Large Language Models

2026-06-12

AI Summary

Current mainstream multilingual large language models employ byte-level Byte Pair Encoding (BPE) tokenizers that exhibit a strong bias toward high-resource languages and Latin scripts, thereby exacerbating inference costs and cross-lingual performance gaps for low-resource languages, particularly in Southeast Asia. This work systematically evaluates multiple fairness-oriented tokenization approaches across 11 Southeast Asian languages under a unified benchmark, assessing their compression efficiency, equity, and downstream task performance through empirical comparisons using a 1.5B-parameter model. We propose two novel methods: Parity-aware BPE, which achieves Pareto optimality between efficiency and fairness, and Morphology-Driven Byte Encoding, which substantially enhances semantic reasoning capabilities. Our findings demonstrate, for the first time, that these objectives can be jointly optimized rather than being fundamentally at odds.

Abstract

Multilingual large language models (LLMs) depend on subword tokenization to bridge discrete text and continuous neural representation. State-of-the-art multilingual LLMs often use Byte-level Byte-Pair Encoding (BPE) tokenizers that structurally favor high-resource languages and Latin scripts. For speakers of underrepresented languages, particularly those across Southeast Asia, this bias inflates inference costs and widens cross-lingual capability gaps. We present the first systematic comparison of equitable tokenizers on a unified benchmark spanning 11 Southeast Asian languages. Beyond tokenizer-level analysis of compression efficiency and cross-lingual equity, we assess downstream task performance through controlled 1.5B-parameter language model training using the same training data. Our results show that Parity-aware BPE lies on the Pareto frontier of the efficiency-equity trade-off, achieving strong compression parity at competitive cost. Morphology-Driven Byte Encoding delivers the best semantic reasoning performance through morphologically richer representations, albeit at a higher computational expense. Byte Latent Transformer underperforms on downstream tasks, possibly because its architectural assumptions misalign with the constraints of limited low-resource training data. Together, our findings demonstrate that cross-lingual fairness and tokenization efficiency are not fundamentally at odds, and offer practical guidance for designing equitable multilingual models.

Problem

• multilingual LLMs
• tokenization bias
• cross-lingual equity
• low-resource languages
• subword tokenization

Innovation

• equitable tokenization
• multilingual LLMs
• cross-lingual fairness
• subword tokenization
• Southeast Asian languages