SampoNLP: A Self-Referential Toolkit for Morphological Analysis of Subword Tokenizers

2026-01-08

arXiv.org

AI Summary

This study addresses the challenge of evaluating subword tokenizers for morphologically rich languages like those in the Uralic family, where high-quality morpheme lexicons are scarce. To circumvent reliance on corpora, the authors propose a method grounded in the Minimum Description Length (MDL) principle, incorporating a self-referential atomicity scoring mechanism that leverages intraword structural cues to filter out compounds and construct high-purity morpheme lexicons. The work delivers the first empirically supported morpheme resources for Finnish, Hungarian, and Estonian, introduces an Integrated Performance Score (IPS) to balance coverage against over-segmentation, and employs elbow-point analysis to recommend optimal BPE vocabulary sizes. Findings highlight the limitations of standard BPE in highly agglutinative languages. The accompanying SampoNLP toolkit and resources are publicly released.

Technology Category

• Natural Language Processing: Lexical Semantics and Morphology
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Semantic Web

Application Category

• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

The quality of subword tokenization is critical for Large Language Models, yet evaluating tokenizers for morphologically rich Uralic languages is hampered by the lack of clean morpheme lexicons. We introduce SampoNLP, a corpus-free toolkit for morphological lexicon creation using MDL-inspired Self-Referential Atomicity Scoring, which filters composite forms through internal structural cues - suited for low-resource settings. Using the high-purity lexicons generated by SampoNLP for Finnish, Hungarian, and Estonian, we conduct a systematic evaluation of BPE tokenizers across a range of vocabulary sizes (8k-256k). We propose a unified metric, the Integrated Performance Score (IPS), to navigate the trade-off between morpheme coverage and over-splitting. By analyzing the IPS curves, we identify the"elbow points"of diminishing returns and provide the first empirically grounded recommendations for optimal vocabulary sizes (k) in these languages. Our study not only offers practical guidance but also quantitatively demonstrates the limitations of standard BPE for highly agglutinative languages. The SampoNLP library and all generated resources are made publicly available: https://github.com/AragonerUA/SampoNLP

Problem

• subword tokenization
• morphological analysis
• Uralic languages
• morpheme lexicon
• tokenizer evaluation

Innovation

• Self-Referential Atomicity Scoring
• Morphological Lexicon Induction
• Subword Tokenization
• Integrated Performance Score
• Low-Resource NLP