Tracing the complexity profiles of different linguistic phenomena through the intrinsic dimension of LLM representations

2026-01-07

arXiv.org

AI Summary

This study investigates how large language models (LLMs) differentially process syntactic complexity—such as coordination versus subordination, center-embedding, and ambiguity resolution—through their internal representations. It introduces intrinsic dimensionality (ID) as a novel metric to quantify linguistic complexity and combines representational similarity analysis, layer ablation, and cross-model comparisons to track ID dynamics across model layers. The findings demonstrate that ID effectively distinguishes between formal and functional aspects of syntactic complexity, with multi-clause constructions eliciting significantly higher ID values. Moreover, distinct types of syntactic complexity activate specific stages of abstract processing, a pattern consistently observed across multiple state-of-the-art LLMs.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We explore the intrinsic dimension (ID) of LLM representations as a marker of linguistic complexity, asking if different ID profiles across LLM layers differentially characterize formal and functional complexity. We find the formal contrast between sentences with multiple coordinated or subordinated clauses to be reflected in ID differences whose onset aligns with a phase of more abstract linguistic processing independently identified in earlier work. The functional contrasts between sentences characterized by right branching vs. center embedding or unambiguous vs. ambiguous relative clause attachment are also picked up by ID, but in a less marked way, and they do not correlate with the same processing phase. Further experiments using representational similarity and layer ablation confirm the same trends. We conclude that ID is a useful marker of linguistic complexity in LLMs, that it allows to differentiate between different types of complexity, and that it points to similar stages of linguistic processing across disparate LLMs.

Problem

• intrinsic dimension
• linguistic complexity
• large language models
• formal complexity
• functional complexity

Innovation

• intrinsic dimension
• linguistic complexity
• large language models
• representational analysis
• syntactic processing