Unsupervised Morphological Tree Tokenizer

2024-06-21

arXiv.org

AI Summary

Conventional statistical subword tokenization methods (e.g., BPE, WordPiece) often split morphemes internally, undermining morphological and semantic integrity. Method: We propose an unsupervised character-level tree-based tokenization framework guided by linguistic morphological structure. First, we introduce MorphOverriding—a mechanism enforcing morpheme atomicity and indivisibility. Second, we design a deep encoder that jointly models intra-word hierarchical structure and contextual representations. Third, we employ self-supervised pretraining coupled with top-down lexical matching for end-to-end tokenization. Contribution/Results: To our knowledge, this is the first fully unsupervised method that automatically induces linguistically plausible vocabulary trees adhering to morphological principles. It significantly outperforms BPE and WordPiece on both morphological segmentation and downstream language modeling tasks, while fully preserving morpheme boundaries and derivational/inflectional hierarchies.

Technology Category

• Natural Language Processing: Lexical Semantics and Morphology
• Machine Learning: Unsupervised & Self-Supervised Learning
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

As a cornerstone in language modeling, tokenization involves segmenting text inputs into pre-defined atomic units. Conventional statistical tokenizers often disrupt constituent boundaries within words, thereby corrupting semantic information. To address this drawback, we introduce morphological structure guidance to tokenization and propose a deep model to induce character-level structures of words. Specifically, the deep model jointly encodes internal structures and representations of words with a mechanism named $ extit{MorphOverriding}$ to ensure the indecomposability of morphemes. By training the model with self-supervised objectives, our method is capable of inducing character-level structures that align with morphological rules without annotated training data. Based on the induced structures, our algorithm tokenizes words through vocabulary matching in a top-down manner. Empirical results indicate that the proposed method effectively retains complete morphemes and outperforms widely adopted methods such as BPE and WordPiece on both morphological segmentation tasks and language modeling tasks. The code will be released later.

Problem

• Improves tokenization by preserving word morphological structure
• Addresses semantic corruption in conventional statistical tokenizers
• Induces character-level structures without annotated training data

Innovation

• Uses morphological structure guidance for tokenization
• Employs MorphOverriding to preserve morpheme indecomposability
• Self-supervised training induces character-level morphological structures