How Tokenization Limits Phonological Knowledge Representation in Language Models and How to Improve Them

2026-04-18

AI Summary

This work addresses the limitation of existing subword tokenization methods, which disregard phonetic information and thereby impair language models’ ability to represent phonological knowledge—such as rhyme patterns and syllable structure. To quantify a tokenizer’s sensitivity to phonology, the authors propose Syllable-Token Alignment Distance (STAD) as a diagnostic metric. They further introduce a lightweight fine-tuning approach guided by International Phonetic Alphabet (IPA) representations, designed to enhance phonological awareness without substantially compromising general-purpose capabilities. Experimental results demonstrate consistent performance gains across three phonology-related tasks, while incurring only minor drops of 1.1% and 0.9% on the GSM8K and MMLU benchmarks, respectively, thus effectively balancing task-specific proficiency with broad linguistic competence.

Technology Category

• Natural Language Processing: Lexical Semantics and Morphology
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Tokenization is the first step in every language model (LM), yet it never takes the sounds of words into account. We investigate how tokenization influences text-only LMs' ability to represent phonological knowledge. Through a series of probing experiments, we show that subword-based tokenization systematically weakens the encoding of both local (e.g., rhyme) and global (e.g., syllabification) phonological features. To quantify this effect, we introduce the syllabification-tokenization alignment distance (STAD), a metric that measures the misalignment between a model's tokenization and the natural syllable boundaries of words, and find that higher misalignment correlates with poorer phonological representations, providing a simple diagnostic for phonology-aware tokenization. To address these limitations, we propose a lightweight IPA-based fine-tuning method that infuses phonological awareness into LMs, leading to consistent improvements across three phonology-related tasks while largely preserving math and general reasoning ability, with 1.1\% and 0.9\% drops on GSM8K and MMLU, respectively.

Problem

• tokenization
• phonological knowledge
• language models
• syllabification
• subword

Innovation

• tokenization
• phonological representation
• syllabification
• IPA-based fine-tuning
• STAD