BabyLM's First Constructions: Causal interventions provide a signal of learning

2025-06-02

AI Summary

This work addresses the open question of whether pre-trained language models (PLMs) possess construction-learning capacity under developmentally plausible conditions—specifically, when trained on input scales approximating infant linguistic exposure. Method: We systematically evaluate BabyLM models—lightweight PLMs trained on developmentally constrained corpora—on their ability to encode form–meaning mappings across diverse syntactic constructions, including surface-similar and semantically sensitive high-difficulty cases. We employ Rozner et al.’s causal intervention analysis framework and a construction-sensitive evaluation protocol. Contribution/Results: All BabyLM models robustly encode multiple constructions; construction encoding strength correlates significantly with downstream task performance. This is the first empirical demonstration that PLMs can acquire constructional knowledge under developmentally realistic data constraints, revealing both structural validity and functional relevance of their grammatical representations.

Technology Category

• Natural Language Processing: Lexical Semantics and Morphology
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Construction grammar posits that children acquire constructions (form-meaning pairings) from the statistics of their environment. Recent work supports this hypothesis by showing sensitivity to constructions in pretrained language models (PLMs), including one recent study (Rozner et al., 2025) demonstrating that constructions shape the PLM's output distribution. However, models under study have generally been trained on developmentally implausible amounts of data, casting doubt on their relevance to human language learning. Here we use Rozner et al.'s methods to evaluate constructional learning in models from the 2024 BabyLM challenge. Our results show that even when trained on developmentally plausible quantities of data, models represent diverse constructions, even hard cases that are superficially indistinguishable. We further find correlational evidence that constructional performance may be functionally relevant: models that better represent constructions perform better on the BabyLM benchmarks.

Problem

• Evaluating constructional learning in models with plausible data
• Assessing relevance of models to human language acquisition
• Linking construction representation to benchmark performance

Innovation

• Causal interventions signal learning in models
• Construction grammar applied to BabyLM challenge
• Models perform better with accurate constructions