How Syntax Specialization Emerges in Language Models

2025-05-26

AI Summary

This study investigates the dynamic emergence mechanisms of syntactic specialization in large language models (LLMs). Addressing the lack of understanding regarding its temporal formation, layer-wise distribution, and influencing factors, we systematically track—throughout the entire training process—the evolving sensitivity of neurons, attention heads, and computational circuits to syntactic structures. We introduce a novel internal consistency metric based on minimal syntactically contrasting sentence pairs, integrated with multi-layer neural activity timing analysis and cross-architecture/initialization controlled experiments. Our findings reveal that syntactic specialization develops progressively, concentrates in deeper layers, and exhibits a model-agnostic “critical period” marked by abrupt functional transitions. We further demonstrate that both model scale and training data volume jointly regulate the pace of syntactic internalization. All results are robust and reproducible; the code, models, and training checkpoints will be publicly released.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: (Computational) Cognitive Architectures

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large language models (LLMs) have been found to develop surprising internal specializations: Individual neurons, attention heads, and circuits become selectively sensitive to syntactic structure, reflecting patterns observed in the human brain. While this specialization is well-documented, how it emerges during training and what influences its development remains largely unknown. In this work, we tap into the black box of specialization by tracking its formation over time. By quantifying internal syntactic consistency across minimal pairs from various syntactic phenomena, we identify a clear developmental trajectory: Syntactic sensitivity emerges gradually, concentrates in specific layers, and exhibits a 'critical period' of rapid internal specialization. This process is consistent across architectures and initialization parameters (e.g., random seeds), and is influenced by model scale and training data. We therefore reveal not only where syntax arises in LLMs but also how some models internalize it during training. To support future research, we will release the code, models, and training checkpoints upon acceptance.

Problem

• How syntactic specialization emerges in language models
• Factors influencing development of syntactic sensitivity
• Tracking syntactic consistency across training stages

Innovation

• Tracking syntactic specialization formation over time
• Quantifying internal syntactic consistency across minimal pairs
• Revealing developmental trajectory influenced by scale and data