PACUTE: Phonology-, Affix-, and Character-level Understanding of Tokens for Filipino

2026-06-13

AI Summary

This study addresses the challenge large language models face in accurately modeling the complex morphological structure of non-concatenative morphological languages like Filipino, where tokenization often misaligns with morpheme boundaries. To this end, the authors introduce PACUTE, the first multi-granularity morphological understanding benchmark for Filipino, comprising 4,600 hierarchical diagnostic tasks spanning morpheme, syllable, and character levels, along with a matching-based scoring mechanism to systematically evaluate both open-source and state-of-the-art commercial models. Experimental results reveal that current models perform near-randomly on morpheme segmentation tasks; even advanced models recognize only a subset of affixes and exhibit substantially lower performance on compositional tasks such as morpheme transformation and syllable segmentation compared to character-level upper bounds, thereby exposing a fundamental bottleneck in contemporary morphological understanding.

Abstract

Large language models (LLMs) process text as sequences of subword tokens, which can obscure the character-level and morphological structure that underlies word formation. This limitation is most acute for languages with non-concatenative morphology, where standard tokenizers systematically misalign token boundaries with morpheme boundaries. We introduce PACUTE, a diagnostic benchmark of 4,600 tasks designed to evaluate morphological understanding in Filipino, a language characterized by productive infixation, reduplication, and diacritic-driven lexical distinctions that are typically absent from written text. PACUTE includes a hierarchical diagnostic framework of six compositional levels that localizes where morphological understanding breaks down. Evaluating open-weight LLMs and frontier commercial models, we find that open-weight models perform near chance on morpheme decomposition regardless of scale. Frontier models perform much better, often recovering individual affixes under contains-match scoring, but remain far below their character-level ceilings on compositional tasks of morpheme transformations and syllabification. These results identify productive morphological composition, rather than character access alone, as the persistent bottleneck for Filipino word-structure understanding.

Problem

• morphological understanding
• non-concatenative morphology
• tokenization
• Filipino language
• subword tokens

Innovation

• morphological understanding
• non-concatenative morphology
• diagnostic benchmark
• Filipino language
• tokenization alignment