MinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignment

2026-06-25

AI Summary

This work addresses the high computational cost and complexity of traditional Unigram tokenizers, which struggle to simultaneously achieve high compression rates and morphological alignment. The authors propose MinGram, a novel approach that initializes with a BPE-derived seed vocabulary and employs a hard EM algorithm optimized over the shortest tokenization paths. Training is streamlined through single-step smoothed score pruning, eliminating the need for suffix arrays, forward–backward passes, and iterative pruning—relying solely on tokenization inference. MinGram jointly optimizes token count as the primary objective and Unigram scores as a secondary criterion. Experiments across six languages demonstrate that MinGram outperforms both BPE and standard Unigram in compression efficiency; its compression-oriented variant matches the strongest token-count-based compressors while preserving superior morphological alignment, and it significantly improves bits-per-byte performance in language modeling.

Abstract

The Unigram tokenizer uses an elegant representation which makes it straightforward to edit vocabularies, but its training is comparatively heavy and complex. We introduce MinGram (Minimalist Unigram), which keeps the token-list representation but simplifies training using a BPE-derived seed vocabulary, Hard EM on a minimum-token path, and a single flat score-pruning step. This removes the suffix array, the forward-backward pass, and the iterative prune loop, leaving a procedure that requires little beyond tokenizer inference itself. By making token count the primary objective and using a Unigram score only as a tiebreak, MinGram keeps the compression of pure token-count methods while retaining much of the morphological alignment and downstream quality of probabilistic ones. Across six languages, MinGram compresses better than both BPE and standard Unigram, and a compression-oriented variant matches the strongest token-count compressors while retaining substantially higher morphological alignment. In controlled downstream language-model training, Unigram-family tokenizers, with MinGram among the best, consistently beat BPE in bits-per-byte.

Problem

• Unigram tokenizer
• tokenization
• compression
• morphological alignment
• vocabulary training

Innovation

• MinGram
• Unigram tokenizer
• tokenization
• compression
• morphological alignment