Emergent retokenization symmetry in large language models: phenomenology and applications

2026-06-13

AI Summary

This study investigates whether large language models exhibit tokenization symmetry—namely, consistent behavior when presented with semantically equivalent inputs that differ only in their tokenization. To this end, the authors propose a retokenization-based perturbation method that systematically evaluates model sensitivity and robustness to tokenization variations without altering the underlying byte sequence. The work reveals, for the first time, that models partially acquire invariance to tokenization redundancy during training and further leverages retokenization as a novel inference-time sampling strategy. Experimental results demonstrate that while this approach may slightly degrade performance on simple tasks, it effectively uncovers correct solutions missed by conventional temperature sampling in complex tasks, substantially enhancing output diversity.

Abstract

Tokenization introduces representational redundancy: under a fixed token vocabulary, every byte string admits many valid token encodings, or segmentations, that decode to the same surface string. However, given a prompt, most language model tokenizers break this representational symmetry by returning a canonical segmentation. Training only on canonical segmentations should influence inference behavior, and there is little reason to expect models to respect segmentation symmetry on downstream tasks. We find that this symmetry partially emerges during training. Here, we probe this emergent symmetry through experiments testing token compositional understanding, representation diversity, and task focused benchmark performance. We primarily use \textbf{retokenization} -- replacing a prompt's canonical tokenization with an alternative segmentation while preserving its bytes exactly. Relative to other prompt perturbations, retokenization is unusually clean because it isolates segmentation effects without changing syntax, semantics or surface form. We use retokenization to study sensitivity and robustness to semantically identical input representations across pretraining and post-training. Moreover, this partial retokenization symmetry suggests a distinct inference-time sampling axis. While temperature sampling generates diverse outputs from the model using its next-token probability distribution, retokenization generates diversity from the model's internal computations through semantically equivalent input representations. We find that while this retokenization sampling strategy can hurt performance on easy problems, it can also recover solutions that conventional sampling does not find. Overall, our work presents retokenization as a simple yet powerful probe of large language models, shedding light on compositional understanding and prompt sensitivity, and offering a novel sampling strategy.

Problem

• retokenization
• tokenization symmetry
• large language models
• prompt sensitivity
• compositional understanding

Innovation

• retokenization
• emergent symmetry
• tokenization invariance
• compositional understanding
• inference-time sampling