An expressivity analysis of hierarchical modelling in deep transformers via bounded-depth grammars

2026-06-16

AI Summary

This work addresses the lack of rigorous theoretical analysis regarding how deep Transformers represent hierarchical linguistic structures. Within a formal framework based on bounded-depth, non-recursive context-free grammars, we provide the first theoretical proof that deep Transformers can encode abstract syntactic states using a linearly increasing number of layers and controlled model size, thereby supporting the hypothesis of linear representability of hierarchical structure. By integrating positional attention mechanisms with residual stream subspace analysis, we construct a Transformer architecture capable of embedding syntactic derivation trees into a low-dimensional linearly separable space, revealing a precise theoretical relationship between the model’s depth and its capacity for hierarchical representation.

Abstract

Deep neural networks are widely believed to derive their expressive power from their ability to form \textbf{hierarchical representations}, capturing progressively more abstract and compositional features across layers. In language modeling, \textbf{transformers} have emerged as the dominant architecture, with early layers capturing local syntactic patterns and later layers encoding more complex clause-level dependencies. While this intuition has shaped model design, there remains a lack of rigorous theoretical work demonstrating \textbf{how} deep transformers represent such hierarchical structures. In this work, we analyze the expressiveness of deep transformer models through the formal lens of bounded-depth, non-recursive context-free grammars. For this class of grammars, we explicitly construct transformers with positional attention whose depth grows linearly with grammar depth, while the neuron count scales with the number of derivation-tree shapes and quadratically with the number of production rules. Our theoretical results support the linear representation hypothesis by demonstrating that these architectures possess the structural capacity to encode abstract grammatical states into low-dimensional, linearly separable subspaces within the residual stream.

Problem

• expressivity
• hierarchical representations
• transformers
• bounded-depth grammars
• theoretical analysis

Innovation

• hierarchical representations
• deep transformers
• bounded-depth grammars
• positional attention
• linear representation hypothesis