Beyond Text Compression: Evaluating Tokenizers Across Scales

2025-06-03

AI Summary

Evaluating tokenizer selection for multilingual large language models lacks efficient, generalizable intrinsic evaluation methods. Method: We propose a lightweight evaluation paradigm based on scaling consistency—demonstrating for the first time that tokenizer performance on small models reliably predicts that on large models—and introduce a novel intrinsic metric grounded in Zipf’s law to enhance cross-lingual downstream task prediction. Contribution/Results: Our multidimensional intrinsic evaluation framework reveals negligible tokenizer impact on English tasks but consistent, language-agnostic performance differences in multilingual settings. The framework achieves >0.85 correlation with unseen-language downstream performance while requiring only 10% of the computational cost of conventional methods. It establishes a reliable, low-cost, cross-lingual benchmark for tokenizer selection, enabling scalable and principled tokenizer evaluation across model scales and languages.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

The choice of tokenizer can profoundly impact language model performance, yet accessible and reliable evaluations of tokenizer quality remain an open challenge. Inspired by scaling consistency, we show that smaller models can accurately predict significant differences in tokenizer impact on larger models at a fraction of the compute cost. By systematically evaluating both English-centric and multilingual tokenizers, we find that tokenizer choice has negligible effects on tasks in English but results in consistent performance differences in multilingual settings. We propose new intrinsic tokenizer metrics inspired by Zipf's law that correlate more strongly with downstream performance than text compression when modeling unseen languages. By combining several metrics to capture multiple aspects of tokenizer behavior, we develop a reliable framework for intrinsic tokenizer evaluations. Our work offers a more efficient path to informed tokenizer selection in future language model development.

Problem

• Evaluating tokenizer quality impacts language model performance
• Assessing tokenizer effects on multilingual versus English tasks
• Developing intrinsic metrics for reliable tokenizer evaluation

Innovation

• Small models predict tokenizer impact efficiently
• New metrics based on Zipf's law improve evaluations
• Combined metrics framework enhances tokenizer selection