Doğal Dil İşlemede Tokenizasyon Standartları ve Ölçümü: Türkçe Üzerinden Büyük Dil Modellerinin Karşılaştırmalı Analizi

2025-08-18

AI Summary

This study addresses the tokenization challenge for morphologically rich, low-resource languages (e.g., Turkish). We propose the first multidimensional evaluation framework tailored to such languages, built upon the newly constructed TR-MMLU dataset. We systematically benchmark mainstream tokenizers across four dimensions: vocabulary size, token count, processing efficiency, and preservation of linguistic structure—introducing two novel metrics: language-specific tokenization ratio (%TR) and token purity (%Pure). Results show that %TR strongly correlates with downstream task performance, whereas %Pure exhibits weak correlation; scaling model parameters alone does not improve linguistic understanding. Crucially, this work provides the first empirical validation that %TR is a more effective indicator of tokenization quality than conventional metrics. It underscores the necessity of linguistically grounded, language-specific tokenization strategies for morphologically complex languages and establishes a reproducible evaluation paradigm and methodological foundation for low-resource NLP.

Technology Category

• Natural Language Processing: Lexical Semantics and Morphology
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Knowledge Representation Languages

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Tokenization is a fundamental preprocessing step in Natural Language Processing (NLP), significantly impacting the capability of large language models (LLMs) to capture linguistic and semantic nuances. This study introduces a novel evaluation framework addressing tokenization challenges specific to morphologically-rich and low-resource languages such as Turkish. Utilizing the Turkish MMLU (TR-MMLU) dataset, comprising 6,200 multiple-choice questions from the Turkish education system, we assessed tokenizers based on vocabulary size, token count, processing time, language-specific token percentages (%TR), and token purity (%Pure). These newly proposed metrics measure how effectively tokenizers preserve linguistic structures. Our analysis reveals that language-specific token percentages exhibit a stronger correlation with downstream performance (e.g., MMLU scores) than token purity. Furthermore, increasing model parameters alone does not necessarily enhance linguistic performance, underscoring the importance of tailored, language-specific tokenization methods. The proposed framework establishes robust and practical tokenization standards for morphologically complex languages.

Problem

• Evaluating tokenization impact on Turkish language models
• Proposing metrics for linguistic structure preservation
• Assessing correlation between tokenization and model performance

Innovation

• Novel evaluation framework for Turkish tokenization
• Language-specific metrics like %TR and %Pure
• Tailored tokenization for morphologically-rich languages