A Distributional Perspective on Word Learning in Neural Language Models

2025-02-09

AI Summary

This work addresses the lack of standardized evaluation protocols for lexical acquisition in language models (LMs) and the unresolved question of whether their learning dynamics resemble human development. We propose the first multidimensional evaluation framework grounded in word distributional properties, specifically quantifying gradient preferences for target words across syntactically and semantically acceptable positions. Methodologically, we construct a distributional signature system integrating distributional modeling, lexical knowledge quantification, and learning trajectory analysis, validated via diagnostic training of small-scale LMs from scratch. Our key contributions are: (1) moving beyond unidimensional metrics to enable fine-grained, decoupled measurement of lexical knowledge; and (2) providing the first empirical evidence that LM lexical acquisition trajectories exhibit no significant correlation with children’s developmental patterns—challenging the “human-like learning” hypothesis. Results indicate that current LMs fail to replicate the stage-wise progression and constraint sensitivity characteristic of human lexical acquisition.

Technology Category

• Natural Language Processing: Lexical Semantics and Morphology
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Language models (LMs) are increasingly being studied as models of human language learners. Due to the nascency of the field, it is not well-established whether LMs exhibit similar learning dynamics to humans, and there are few direct comparisons between learning trajectories in humans and models. Word learning trajectories for children are relatively well-documented, and recent work has tried to extend these investigations to language models. However, there are no widely agreed-upon metrics for word learning in language models. We take a distributional approach to this problem, defining lexical knowledge in terms of properties of the learned distribution for a target word. We argue that distributional signatures studied in prior work fail to capture key distributional information. Thus, we propose an array of signatures that improve on earlier approaches by capturing knowledge of both where the target word can and cannot occur as well as gradient preferences about the word's appropriateness. We obtain learning trajectories for a selection of small language models we train from scratch, study the relationship between different distributional signatures, compare how well they align with human word learning trajectories and interpretable lexical features, and address basic methodological questions about estimating these distributional signatures. Our metrics largely capture complementary information, suggesting that it is important not to rely on a single metric. However, across all metrics, language models' learning trajectories fail to correlate with those of children.

Problem

• Compare word learning in language models and humans.
• Develop metrics for assessing word learning in models.
• Analyze alignment between model and child learning trajectories.

Innovation

• Distributional approach to word learning
• Improved distributional signature metrics
• Comparison of model and human trajectories