Tokens with Meaning: A Hybrid Tokenization Approach for NLP

2025-08-19

AI Summary

Existing subword tokenization methods (e.g., BPE, WordPiece) rely heavily on surface-form frequency statistics, rendering them ill-suited for morphologically rich, agglutinative languages. To address this, we propose a hybrid tokenization framework integrating linguistic rules with statistical learning: (1) phonemic normalization mitigates orthographic variation; (2) an explicit root–affix dictionary models morphological structure; (3) a shared identifier mechanism balances morpheme fidelity and subword efficiency; and (4) unified special tokens handle whitespace and case, curbing vocabulary bloat. Evaluated on the TR-MMLU benchmark, our method achieves 90.29% tokenization accuracy and 85.8% pure segmentation rate for Turkish—substantially outperforming LLaMA, Gemma, and GPT tokenizers. This work represents the first systematic integration of phonological, morphological, and statistical modeling in large-scale pretraining tokenization, significantly enhancing cross-lingual semantic consistency and out-of-vocabulary generalization—particularly for agglutinative languages.

Technology Category

• Natural Language Processing: Lexical Semantics and Morphology
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Tokenization plays a pivotal role in natural language processing (NLP), shaping how text is segmented and interpreted by language models. While subword methods such as Byte Pair Encoding (BPE) and WordPiece have been effective, they often struggle with morphologically rich and agglutinative languages because they rely on frequency rather than linguistic structure. We introduce a hybrid tokenization framework that combines rule-based morphological analysis with statistical subword segmentation. The method uses phonological normalization, root-affix dictionaries, and a novel algorithm that balances morpheme preservation with vocabulary efficiency. It assigns shared identifiers to phonologically variant affixes (e.g., -ler and -lar) and altered root forms (e.g., kitap vs. kitabı), reducing redundancy while maintaining semantic integrity. Special tokens are added for whitespace and case, including an UPPERCASE marker to avoid vocabulary inflation from capitalization. BPE is integrated for out-of-vocabulary coverage without harming morphological coherence. On the TR-MMLU benchmark, the tokenizer achieves the highest Turkish Token Percentage (90.29%) and Pure Token Percentage (85.8%). Comparisons with tokenizers from LLaMA, Gemma, and GPT show more linguistically meaningful and coherent tokens. Although demonstrated on Turkish, the approach is language-independent and adaptable to other languages, offering a practical path toward more interpretable and effective multilingual NLP systems.

Problem

• Improving tokenization for morphologically rich languages
• Combining linguistic rules with statistical subword segmentation
• Reducing vocabulary redundancy while preserving semantic meaning

Innovation

• Hybrid tokenization combining morphological analysis with subword segmentation
• Phonological normalization and root-affix dictionaries for semantic integrity
• Integration of BPE for out-of-vocabulary coverage without morphological harm