When transformers learn "impossible" languages, what do they learn?

2026-06-29

AI Summary

This study investigates why Transformer-based language models struggle to learn artificial “impossible” languages—those not attested in human linguistic systems—by distinguishing between two competing hypotheses: insufficient grammatical sensitivity versus inherent limitations in generative capacity. Using GPT-2–style models trained on perturbed variants of English, the authors evaluate grammaticality judgment via BLiMP minimal pairs and assess long-form generation quality. The work establishes, for the first time, a direct link between the unlearnability of impossible languages and model-specific generative deficits. Results show that while models retain sensitivity to local grammatical violations, they exhibit significant degradation in long-range coherence during generation. These findings support the hypothesis that generative limitations and error propagation—not mere insensitivity to syntax—underlie the absence of such languages in human linguistic evolution.

Abstract

Recent work suggests that transformer language models show a bias towards human languages over unnatural ("impossible") languages argued to be unacquirable by humans. However, this literature has largely based these claims on differences in sample efficiency and test-set perplexity, rather than on direct evaluations of the linguistic capacities that could plausibly explain non-attestation in human languages. We evaluate two theoretically motivated linking hypotheses: impossibility arising from deficiencies in grammatical sensitivity or generative production. Using GPT-2 style models trained on perturbed "impossible" variants of English, we measure sensitivity to grammaticality using BLiMP minimal pairs, finding that model performance exhibits only gradual degradation, mediated by the language's information locality. In contrast, these models exhibited pronounced failures in generation, producing substantially fewer high-quality sentences at longer lengths. Together, these results suggest generative deficiency and transmission failures as a plausible linking hypothesis between language model behaviour and non-attestation of impossible languages.

Problem

• impossible languages
• grammatical sensitivity
• generative production
• language non-attestation
• transformer language models

Innovation

• generative deficiency
• grammatical sensitivity
• impossible languages
• language transmission
• transformer language models