Diverging Transformer Predictions for Human Sentence Processing: A Comprehensive Analysis of Agreement Attraction Effects
🤖 AI Summary
This study investigates whether current autoregressive Transformer models can effectively emulate the human cognitive mechanisms underlying morphosyntactic agreement processing, with a particular focus on the asymmetric interference patterns observed in agreement attraction phenomena. Within a surprisal-based theoretical framework, the authors systematically evaluate the predictive performance of eleven Transformer models—varying in scale and architecture—across multiple English agreement attraction constructions, comparing model predictions against human reading-time data. This work presents the first comprehensive, multi-model assessment of cognitive plausibility across a full range of agreement attraction configurations. Results reveal that while existing models perform reasonably well on prepositional phrase constructions, their accuracy markedly declines in object-extracted relative clauses, and they consistently fail to replicate the human-specific asymmetry in interference effects, thereby exposing fundamental limitations in their capacity to model complex syntactic dependencies.
Application Category
📝 Abstract
Transformers underlie almost all state-of-the-art language models in computational linguistics, yet their cognitive adequacy as models of human sentence processing remains disputed. In this work, we use a surprisal-based linking mechanism to systematically evaluate eleven autoregressive transformers of varying sizes and architectures on a more comprehensive set of English agreement attraction configurations than prior work. Our experiments yield mixed results: While transformer predictions generally align with human reading time data for prepositional phrase configurations, performance degrades significantly on object-extracted relative clause configurations. In the latter case, predictions also diverge markedly across models, and no model successfully replicates the asymmetric interference patterns observed in humans. We conclude that current transformer models do not explain human morphosyntactic processing, and that evaluations of transformers as cognitive models must adopt rigorous, comprehensive experimental designs to avoid spurious generalizations from isolated syntactic configurations or individual models.
Problem
Research questions and friction points this paper is trying to address.
transformer
sentence processing
agreement attraction
cognitive modeling
morphosyntactic processing
Innovation
Methods, ideas, or system contributions that make the work stand out.
agreement attraction
transformer models
surprisal
sentence processing
cognitive modeling
🔎 Similar Papers
No similar papers found.
