Phonemes to the Rescue: Multilingual Tokenization Based on International Phonetic Alphabet

2026-06-18

AI Summary

This work addresses the cross-lingual performance imbalance in multilingual language models, which arises from biases toward high-resource languages and disparities in character-to-byte ratios during tokenization. To mitigate these issues, the study systematically introduces the International Phonetic Alphabet (IPA) as a language-agnostic input representation and constructs an IPA-based subword tokenizer spanning 24 languages and 14 writing systems. This approach substantially reduces the symbol vocabulary size, enhances cross-lingual character overlap, and balances byte-length distributions. Consequently, it significantly improves tokenization quality for non-Latin scripts and demonstrates stronger generalization capabilities on unseen languages and writing systems.

Abstract

Multilingual language models often exhibit performance disparities across languages that can arise as early as the tokenization stage. Widely-used subword tokenization approaches favor high-resource languages, and tokenizer-free methods still yield longer sequences for scripts with a higher bytes-per-character ratio. To address these shortcomings, we propose to use the International Phonetic Alphabet (IPA) as a language-agnostic input representation for multilingual tokenizers. IPA provides a compact symbol inventory, greater cross-lingual character overlap, and a more balanced byte-per-character distribution across languages. We train matched pairs of text vs. IPA subword tokenizers across 24 languages and 14 scripts and demonstrate that IPA tokenizers consistently improve tokenization quality, especially for non-Latin scripts, and generalize more effectively to unseen languages and scripts.

Problem

• multilingual tokenization
• performance disparity
• subword tokenization
• language bias
• bytes-per-character ratio

Innovation

• International Phonetic Alphabet
• multilingual tokenization
• subword tokenization
• language-agnostic representation
• cross-lingual generalization