Explicit Boundary Markers for Subword Vocabularies

2026-08-09

AI Summary

This work addresses the issue of embedding fragmentation and inconsistency in traditional subword tokenizers, which arises from their implicit handling of whitespace and case variants, leading to redundant segmentation of identical words. To resolve this, the authors propose a novel tokenization scheme that replaces implicit space conventions with explicit word boundary markers—using paired tokens to represent inter-word spaces—and introduces a case-shift coding mechanism to uniformly encode case variants, thereby enabling shared internal representations across different forms of the same word. Integrated into both subword learning and language modeling pipelines, the method achieves character-per-token compression rates comparable to baselines (within 1%) across six languages, while significantly reducing bits-per-byte on downstream language modeling tasks, consistently outperforming existing approaches.

Abstract

Subword tokenizers represent many common words twice in space-using writing systems, once with a leading space and once without. The two entries have separate embeddings in models, so occurrences of one word are divided across rows that are trained independently, and the two forms need not even segment the string the same way: " together" may be a single entry while the same word without a preceding space is tokenized as "to|gether". Capitalization divides a word further, into as many as six forms. We introduce an alternative to standard whitespace conventions using an explicit word boundary marker, which prevents such duplication. Words are delimited by the boundary markers, and spaces between words are represented as pairs of such markers. Two shift codes do the same for title case and upper case, allowing one internal representation of a word to be re-used across different settings. Switching to this convention mitigates the duplicate-entry issue, but does not improve tokenization compression: for both vocabulary-learning algorithms, the best marker scheme stays within one percent of the baseline in characters per token, averaged across six languages. It does result in better language modeling performance. Every marker scheme tested downstream reaches lower bits per byte than the baseline, suggesting that duplication carries a cost that compression does not capture.

Problem

• subword tokenization
• word duplication
• boundary markers
• vocabulary redundancy
• language modeling

Innovation

• explicit boundary markers
• subword tokenization
• word duplication
• case normalization
• language modeling