Beyond Perplexity: UTF-8 Validity in Byte-aware Language Models

2026-06-12

AI Summary

This work addresses the challenge that byte-level language models often generate invalid UTF-8 sequences when producing rare or previously unseen characters, thereby compromising the reliability of multilingual text generation. The authors train a 355-million-parameter, byte-level language model on 80 billion tokens of multilingual data and introduce a perplexity-independent evaluation protocol to assess structural validity of UTF-8 sequences. Their findings reveal that achieving convergence in UTF-8 validity requires approximately twice as much training data as needed for perplexity stabilization—42 billion versus 21 billion tokens. Notably, in context-free generation, rare characters exhibit higher structural validity than common ones, challenging conventional assumptions in representation learning. These results demonstrate that reliable UTF-8 sequence generation constitutes a distinct modeling capability beyond mere perplexity minimization.

Abstract

Byte-level tokenization enables language models to handle any Unicode input, but models can generate invalid UTF-8 sequences when encountering rare or unseen characters. We investigate the relationship between training scale and UTF-8 generation reliability with a 355M parameter model trained on 80B tokens from a balanced multilingual corpus of English, Japanese, Korean, and Chinese. We introduce multiple evaluation protocols that isolate UTF-8 structural validity from language modeling. UTF-8 validity convergence lags perplexity by a roughly a factor of two: perplexity stabilizes after 2.1B tokens, but UTF-8 validity requires 4.2B tokens. In context-free generation, rare characters achieve higher structural validity than common characters, suggesting over-specialization of frequent character representations. Through experiments, we observed that reliable UTF-8 generation is a distinct capability requiring evaluation beyond perplexity.

Problem

• UTF-8 validity
• byte-level tokenization
• language models
• Unicode
• rare characters

Innovation

• UTF-8 validity
• byte-level tokenization
• language model evaluation
• multilingual modeling
• training scale