A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformers

2025-03-05

AI Summary

This work investigates how logarithmic depth growth—Θ(log n) in input length n—enhances the expressive power of Transformer models, specifically for state tracking and multi-step reasoning over short-to-medium-length sequences. Method: We combine theoretical analysis—including AC⁰ complexity lower bounds, circuit simulation, and attention-based modeling—with empirical validation. Contribution/Results: We establish, for the first time, that Θ(log n) depth suffices for Transformers to recognize regular languages and decide graph connectivity—two classical tasks provably intractable for fixed-depth Transformers. Our results demonstrate that logarithmic depth scaling improves reasoning efficiency more effectively than width expansion or naive layer stacking. Moreover, the theoretically derived minimal depth closely matches empirically observed training requirements, providing a verifiable, theoretically grounded design principle for lightweight sequence reasoning models.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Machine Learning: Deep Neural Architectures and Foundation Models
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Recent theoretical results show transformers cannot express sequential reasoning problems over long input lengths, intuitively because their computational depth is bounded. However, prior work treats the depth as a constant, leaving it unclear to what degree bounded depth may suffice for solving problems over short inputs, or how increasing the transformer's depth affects its expressive power. We address these questions by analyzing the expressive power of transformers whose depth can grow minimally with context length $n$. We show even highly uniform transformers with depth $Theta(log n)$ can express two important problems: recognizing regular languages, which captures state tracking abilities, and graph connectivity, which underlies multi-step reasoning. Notably, both of these problems cannot be expressed by fixed-depth transformers under standard complexity conjectures, demonstrating the expressivity benefit of growing depth. Moreover, our theory quantitatively predicts how depth must grow with input length to express these problems, showing that depth scaling is more efficient than scaling width or chain-of-thought steps. Empirically, we find our theoretical depth requirements for regular language recognition match the practical depth requirements of transformers remarkably well. Thus, our results clarify precisely how depth affects transformers' reasoning capabilities, providing potential practical insights for designing models that are better at sequential reasoning.

Problem

• Analyzes expressive power of transformers with depth growing with context length.
• Shows log-depth transformers can solve regular language recognition and graph connectivity.
• Demonstrates depth scaling is more efficient than width or chain-of-thought steps.

Innovation

• Transformers with depth scaling as log(n)
• Express regular languages and graph connectivity
• Depth scaling more efficient than width scaling