To Each Language Its Tokenizer: Modular Tokenizers for Efficient Multilingual LLMs

2026-09-14

AI Summary

本文针对多语言大模型的词汇压缩不均和资源浪费问题，提出模块化分词器学习方法及预训练策略，提高效率与跨语言公平性。

Abstract

Multilingual Large Language Models (LLMs) traditionally rely on a single vocabulary shared by all supported languages, which can lead to uneven compression across them. Moreover, their large embedding and output matrices increase memory usage and slow inference, notably for small-scale models. It is also wasteful as models are often used for only a subset of languages. To address these issues, we introduce a modular framework for multilingual model training. First, we propose methods to learn large modular BPE and Unigram tokenizers that enable extraction of subtokenizers tailored to any language subset. These subtokenizers achieve compression on par with monolingual tokenizers and improve cross-lingual fairness. Second, we design a pretraining strategy that samples subtokenizers to form batches, restricting predictions to the relevant vocabulary subset and allowing efficient training despite a large vocabulary. This supports efficient inference with any combination of language-specific vocabularies. Therefore, it reduces memory usage and speeds up inference in models without sacrificing performance.

Problem

• Multilingual Large Language Models
• vocabulary compression
• memory usage
• inference speed

Innovation

• modular tokenizers
• multilingual LLMs
• efficient training
• cross-lingual fairness
• memory usage reduction