Compute Optimal Tokenization

2026-05-01

AI Summary

This study investigates how the information granularity of data units (tokens) influences the scaling laws of language models, with a focus on the role of compression rate—defined as average bytes per token—in determining compute-optimal configurations. By training 988 latent tokenization models (BLTs) with flexible compression rates across model sizes from 50M to 7B parameters, and validating across multilingual and subword tokenization settings, the authors find that under compute-optimal conditions, model size should scale proportionally with data volume measured in bytes rather than tokens. Moreover, the optimal compression rate is substantially lower than that achieved by standard byte-pair encoding (BPE) and decreases further as available compute budget diminishes. These findings offer a byte-centric tokenization strategy for designing more efficient language models.

Abstract

Scaling laws enable the optimal selection of data amount and language model size, yet the impact of the data unit, the token, on this relationship remains underexplored. In this work, we systematically investigate how the information granularity of tokens, controlled by the compression rate (i.e., average bytes of text per token), affects scaling trends. We train 988 latent tokenized models (BLT) ranging from 50M to 7B parameters that enable setting the desired compression rate. This flexibility allows us to study the role of compression rate well beyond 4.57 bytes per token obtained with a popular BPE tokenizer. Our experiments reveal that in compute-optimal configurations, model parameter counts scale proportionally to data size measured in bytes, not in tokens as commonly perceived (Kaplan et al., 2020; Hoffmann et al., 2022). Furthermore, we discover that the optimal compression rate differs from the one obtained with BPE and decreases with compute. These findings generalize to both latent and subword tokenization, as well as to languages other than English, guiding language model developers on tokenization scheme selection for maximal compute efficiency.

Problem

• tokenization
• scaling laws
• compression rate
• compute efficiency
• language models

Innovation

• compression rate
• scaling laws
• tokenization
• compute-optimal
• latent tokenization