In-Place Tokenizer Expansion for Pre-trained LLMs

2026-07-16

AI Summary

This work addresses the inefficiency of fixed vocabularies in pretrained large language models when supporting new languages, which leads to tokenization redundancy, increased latency, and higher energy consumption—particularly hindering on-device multilingual capabilities. The authors propose an in-place tokenizer expansion method that preserves the original vocabulary while extending it through continued application of Byte Pair Encoding (BPE) merge rules on multilingual corpora. New subword embeddings are initialized as the mean of their constituent subword embeddings, followed by a two-stage training process involving embedding fine-tuning and full-model continued pretraining to recover performance. This approach achieves the first lossless, in-place tokenizer expansion for already-trained large models, ensuring new tokens decompose precisely into atomic units. Evaluated on the LFM2-8B-A1B model with a 128K-token vocabulary, the method reduces token counts for Hindi and Vietnamese by 2.4–2.6× (up to 4× for Thai), yielding an estimated 2.2–3.7× speedup in character-level decoding. The extended model and tokenizer are publicly released.

Abstract

A tokenizer fixed at the start of pre-training allocates vocabulary in proportion to the pre-training corpus, reflecting the deployment priorities at that time. When those priorities shift, languages added later are split into many more tokens per word, which can raise latency, compute, and energy consumption for users of those languages. Cloud models can afford a broad vocabulary because the embedding and LM-head matrices are a small fraction of their parameters. On a compact model those matrices are a material share of per-token decode bandwidth, so on-device models ship small vocabularies and accept fragmentation outside a fixed language set. We present tokenizer expansion, an in-place recipe for upgrading a pre-trained model's tokenizer when the model producer controls its design. We continue the existing tokenizer's BPE merges on a multilingual corpus, so most source tokens carry over unchanged as single tokens and every new token has an exact decomposition into source tokens. We copy the carried-over embedding rows unchanged and initialize new rows as the mean of their source sub-token embeddings. A two-stage adaptation, embedding-only training then full-model continued pre-training, recovers source-checkpoint quality. We apply the recipe to a continued pre-trained checkpoint of LFM2-8B-A1B, an 8B-parameter Mixture-of-Experts model, to help produce LFM2.5-8B-A1B with a 128K tokenizer. The expanded tokenizer encodes Hindi and Vietnamese in roughly $2.4\times$ and $2.6\times$ fewer tokens than the source (up to $4.0\times$ on Thai). Combining these reductions with the measured per-token cost of the larger vocabulary, we estimate a $2.2$-$3.7\times$ per-character decode speedup for these languages across our reference devices. We release the model weights and the expanded tokenizer, and report the negative findings that shaped the recipe.

Problem

• tokenizer expansion
• pre-trained LLMs
• vocabulary fragmentation
• multilingual efficiency
• on-device inference

Innovation

• tokenizer expansion
• in-place adaptation
• multilingual BPE
• embedding initialization
• on-device LLM