Subword models struggle with word learning, but surprisal hides it

2025-02-18

AI Summary

Subword tokenization models conflate lexical and syntactic learning, contradicting psycholinguistic evidence that lexical acquisition precedes syntactic development. Method: We conduct psycholinguistic lexical decision tasks, perplexity analysis, and cross-granularity (subword vs. character-level) comparative experiments on neural language models. Contribution/Results: Character-level models consistently achieve high accuracy in distinguishing words from nonwords, whereas subword models perform significantly worse. Crucially, we provide the first empirical demonstration that lexical and syntactic learning are temporally and mechanistically separable in character models—lexical representations emerge earlier and independently—but remain tightly coupled and inseparable in subword models. These findings challenge the dominant subword modeling paradigm and establish character-level modeling as a cognitively more plausible computational framework for language acquisition.

Technology Category

• Natural Language Processing: Lexical Semantics and Morphology
• Cognitive Modeling & Cognitive Systems: Symbolic Representations
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Models for Web evolution
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We study word learning in subword and character language models with the psycholinguistic lexical decision task. While subword LMs struggle to discern words and non-words with high accuracy, character LMs solve this task easily and consistently. Furthermore, when comparing word learning and syntactic learning, both processes are separable in character LM where word learning predates syntactic learning, whereas these processes are simultaneous in subword LM. This raises questions about the adequacy of subword LMs for modeling language acquisition and positions character LMs as a viable alternative.

Problem

• Subword models struggle with word learning.
• Character models excel in lexical decision tasks.
• Subword and character models differ in learning processes.

Innovation

• Character language models excel
• Separate word and syntactic learning
• Challenge subword model adequacy