A Pilot Study of Autocompleting Tokenizers

2026-08-15

AI Summary

This study addresses the computational bottlenecks and performance degradation caused by verbose sequences in byte-level tokenization. We propose a lightweight autoregressive input compression mechanism based on context prediction, which achieves implicit information representation by removing predictable bytes. This approach significantly shortens effective sequence lengths while preserving translation quality. Experiments demonstrate that English-French translation sequences are reduced by nearly one-third without performance loss, while multilingual tasks achieve compression rates of 0.47–0.67 with maintained or improved translation quality. By effectively reducing the computational overhead of byte-level models, this work substantially enhances efficiency in multilingual machine translation.

Abstract

Modern input methods routinely rely on autocomplete to omit information that can be recovered from local context. Inspired by these autocomplete-assisted writing systems, we investigate whether Transformer inputs can be compressed in a similar manner. Byte-level tokenization offers a simple and language-independent alternative to subword tokenization, but its longer input sequences typically result in increased computational cost and reduced model quality. We propose a compression scheme that employs a lightweight autoregressive byte language model to identify and remove bytes that are easily predictable from their surrounding context before Transformer processing. The resulting compressed representation is then provided as input to a standard encoder--decoder Transformer. Experiments on machine translation show that a substantial fraction of source-language bytes can be omitted without degrading translation quality. On English--French, our best method preserves translation performance while reducing source sequence length by nearly one-third. Additional experiments on Finnish--English, Russian--English, and Chinese--English demonstrate that the approach generalizes across diverse writing systems and morphological typologies, yielding comparable or improved translation quality at compression ratios between 0.47 and 0.67. These findings suggest that many input bytes are predictable enough to be represented implicitly rather than explicitly, providing a simple mechanism for reducing the sequence-length overhead associated with byte-level models.

Problem

• Byte-level tokenization
• Input compression
• Sequence length
• Computational cost

Innovation

• Autocompleting Tokenizers
• Byte-level Compression
• Autoregressive Language Model
• Input Compression
• Machine Translation