Word Recovery in Large Language Models Enables Character-Level Tokenization Robustness

2026-03-11

AI Summary

Large language models exhibit surprising robustness to character-level non-standard inputs, yet the underlying mechanism remains unclear. This work proposes the first “word restoration” hypothesis, suggesting that models reconstruct canonical word-level representations through attention among characters belonging to the same word in early layers. To substantiate this claim, we employ a suite of interpretability techniques—including decoding-based hidden state analysis, subspace intervention, and fine-grained attention masking—to provide causal evidence: removing the word restoration–associated subspace or masking critical attention patterns significantly degrades model performance. Our experiments demonstrate that word restoration is a core mechanism underpinning character-level robustness, offering deeper insight into the model’s tokenization invariance.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Large language models (LLMs) trained with canonical tokenization exhibit surprising robustness to non-canonical inputs such as character-level tokenization, yet the mechanisms underlying this robustness remain unclear. We study this phenomenon through mechanistic interpretability and identify a core process we term word recovery. We first introduce a decoding-based method to detect word recovery, showing that hidden states reconstruct canonical word-level token identities from character-level inputs. We then provide causal evidence by removing the corresponding subspace from hidden states, which consistently degrades downstream task performance. Finally, we conduct a fine-grained attention analysis and show that in-group attention among characters belonging to the same canonical token is critical for word recovery: masking such attention in early layers substantially reduces both recovery scores and task performance. Together, our findings provide a mechanistic explanation for tokenization robustness and identify word recovery as a key mechanism enabling LLMs to process character-level inputs.

Problem

• tokenization robustness
• word recovery
• large language models
• character-level tokenization
• mechanistic interpretability

Innovation

• word recovery
• tokenization robustness
• mechanistic interpretability
• character-level tokenization
• in-group attention