Language Re-generation: An investigation into information locality effects on reconstruction

2026-07-11

AI Summary

This study investigates whether language models can reconstruct natural language from “impossible languages” that violate the principle of information locality, thereby revealing their inductive biases. By fine-tuning GPT-2 on three types of perturbed data—including globally shuffled text—and evaluating performance through syntactic analysis (Triple F1), exact match accuracy, and controlled sentence-length experiments, the work quantitatively demonstrates, for the first time, a strong architectural bias toward information locality. The findings show that reconstruction difficulty increases with greater disruption of locality; long sentences are recoverable only when local structure is preserved; reconstructed texts exhibit shorter dependency distances; and structural recovery, surface-form recovery, and fluency can be decoupled. Crucially, reconstruction difficulty aligns closely with learnability, establishing a systematic link between the two.

Abstract

Information locality, the tendency for syntactically related words to appear close together, shapes both human language processing and language model learning. While prior work has examined whether language models can acquire impossible languages, it remains unclear whether they can recover natural language from such input and what this reveals about their inductive biases. We address this by complementing learnability-based approaches with a reconstruction framework: fine-tuning GPT-2 models pre-trained on impossible languages to reconstruct natural English from three perturbation types. Our findings show that the recovered structures exhibit shorter dependency lengths than the original text, mirroring the locality preference observed in unconstrained language model generation and providing a quantitative signature of an architectural bias that learnability experiments alone do not reveal. Recovery difficulty increases with the degree of locality disruption. Structural recovery (dependency Triple F1) dissociates from surface recovery (Exact Match), while fluency dissociates from faithful reconstruction under global shuffling. Sentence length further modulates performance: longer sentences facilitate recovery when local structure is preserved but lead to complete collapse under global shuffling. Finally, recovery difficulty tracks learnability difficulty across perturbation types, suggesting that information locality is the shared constraint governing both.

Problem

• information locality
• language reconstruction
• inductive bias
• dependency length
• learnability

Innovation

• information locality
• language re-generation
• inductive bias
• dependency length
• reconstruction framework