Heterogeneity in Formal Linguistic Competence of Language Models: Is Data the Real Bottleneck?

2026-04-20

AI Summary

This study addresses the puzzling underperformance of large language models on certain formal syntactic tasks, where accuracy falls significantly below chance levels—a phenomenon whose origin remains unclear. By injecting only 1% targeted synthetic data into the pretraining corpus, the authors demonstrate on GPT-2 Small and a FineWeb subset that this deficiency stems primarily from data scarcity rather than architectural limitations. Experimental results show marked improvements in 8 out of the 9 worst-performing BLiMP paradigms—for instance, accuracy on the only_npi_scope task rises from 20.9% to 69.4%—without degrading overall linguistic competence. This work is the first to reveal, through minimal data intervention, the critical role of training data composition in shaping syntactic capabilities, offering a novel pathway toward enhancing grammatical understanding in language models.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Large Language Models (LLMs) exhibit a puzzling disparity in their formal linguistic competence: while they learn some linguistic phenomena with near-perfect mastery, they often perform below chance on others, even after training on trillions of tokens. In this work, we investigate whether these failures stem from inherent architectural limitations or simply the scarcity of these specific grammatical constructions in web-scale corpora. We pre-train simple GPT-2 Small (124M) models on a 100M-token random sample of the FineWeb corpus and intervene by injecting a minimal amount (1%) of synthetic data targeting specific linguistic phenomena. We find that this targeted intervention substantially improves model performance in 8 out of the 9 worst-performing BLiMP paradigms - notably the accuracy on a specific paradigm, only_npi_scope, surges from 20.9% to 69.4%. Furthermore, we observe that these interventions generally preserve or slightly improve aggregate performance. However, while we also identify a resistant phenomenon, principle_A_c_command, whose performance remains below chance even after our data augmentation, our findings do serve as an optimistic existence proof that even small language models can substantially improve on those linguistic phenomena on which models typically perform poorly, provided the pre-training data contains sufficient exposure to them. This suggests that efforts towards human-scale language modeling may benefit greatly by focusing on data composition. The code to reproduce our results is open-sourced at https://github.com/kowndinya-renduchintala/heterogeneity-in-formal-linguistic-competence.

Problem

• linguistic competence
• language models
• data scarcity
• grammatical constructions
• model heterogeneity

Innovation

• data augmentation
• linguistic competence
• synthetic data intervention
• BLiMP
• language model heterogeneity