Disjoint Processing Mechanisms of Hierarchical and Linear Grammars in Large Language Models

2025-01-15

AI Summary

It remains unclear whether large language models (LLMs) intrinsically develop dissociated hierarchical versus linear syntactic processing mechanisms—and whether such dissociation arises independently of semantics and lexical frequency distributions. Method: We conduct cross-lingual and nonce-word syntactic generation, targeted neuron ablation, and cross-linguistic structural behavioral analysis to causally isolate hierarchical and linear syntactic components in LLMs. Contribution/Results: We identify and localize hierarchical-selective neural circuits that remain robustly activated even on semantically null, frequency-uninformative nonce syntax—demonstrating that hierarchical sensitivity constitutes an inherent inductive bias, not a byproduct of semantic or statistical cues. This provides the first causal evidence for modular, compositionally structured syntactic representations in LLMs, revealing how grammatical competence emerges intrinsically during large-scale pretraining.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

All natural languages are structured hierarchically. In humans, this structural restriction is neurologically coded: when two grammars are presented with identical vocabularies, brain areas responsible for language processing are only sensitive to hierarchical grammars. Using large language models (LLMs), we investigate whether such functionally distinct hierarchical processing regions can arise solely from exposure to large-scale language distributions. We generate inputs using English, Italian, Japanese, or nonce words, varying the underlying grammars to conform to either hierarchical or linear/positional rules. Using these grammars, we first observe that language models show distinct behaviors on hierarchical versus linearly structured inputs. Then, we find that the components responsible for processing hierarchical grammars are distinct from those that process linear grammars; we causally verify this in ablation experiments. Finally, we observe that hierarchy-selective components are also active on nonce grammars; this suggests that hierarchy sensitivity is not tied to meaning, nor in-distribution inputs.

Problem

• Complex Sentence Processing
• Language Models
• Linguistic Habit Irrelevance

Innovation

• Large Language Models
• Complex Sentence Processing
• Grammar Learning