Which Word Orders Facilitate Length Generalization in LMs? An Investigation with GCG-Based Artificial Languages

2025-10-14

AI Summary

This study investigates whether language models (LMs) possess typology-driven inductive biases—specifically, whether they more readily acquire and generalize to typologically frequent word orders (e.g., SOV, SVO) than to rare or attested-violating structures. Method: The authors construct synthetic languages grounded in Generalized Categorial Grammar (GCG), enabling modeling of non-local dependencies and mild context-sensitivity while systematically controlling word-order typology. Transformer models are trained and evaluated on length generalization under rigorously isolated experimental conditions. Contribution/Results: Typological frequency robustly predicts model generalization difficulty: common word orders yield up to a 2.3× improvement in length extrapolation success rate over rare ones. This provides the first typological evidence for structural inductive biases in LMs. Moreover, the work establishes GCG as a novel, controllable probe framework for modeling complex syntactic phenomena in neural language modeling.

Technology Category

• Natural Language Processing: Lexical Semantics and Morphology
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Analogy

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Whether language models (LMs) have inductive biases that favor typologically frequent grammatical properties over rare, implausible ones has been investigated, typically using artificial languages (ALs) (White and Cotterell, 2021; Kuribayashi et al., 2024). In this paper, we extend these works from two perspectives. First, we extend their context-free AL formalization by adopting Generalized Categorial Grammar (GCG) (Wood, 2014), which allows ALs to cover attested but previously overlooked constructions, such as unbounded dependency and mildly context-sensitive structures. Second, our evaluation focuses more on the generalization ability of LMs to process unseen longer test sentences. Thus, our ALs better capture features of natural languages and our experimental paradigm leads to clearer conclusions -- typologically plausible word orders tend to be easier for LMs to productively generalize.

Problem

• Investigating word orders enabling LM length generalization
• Using GCG-based artificial languages for broader coverage
• Testing typologically plausible word order generalization ability

Innovation

• Adopts Generalized Categorial Grammar for artificial languages
• Evaluates generalization to unseen longer test sentences
• Tests typologically plausible word orders in models