InfoFlow: A Framework for Multi-Layer Transformer Analysis

2026-05-18

AI Summary

Existing theory lacks a rigorous characterization of the approximation capabilities of multilayer Transformers and their parameter efficiency advantages over single-layer models. This work proposes the InfoFlow framework, which formalizes the information propagation mechanism in deep Transformer architectures by modeling the set of input information accessible to each token at every layer, and establishes a quantitative relationship between this mechanism and function approximation rates. The framework reveals, for the first time, that in certain retrieval tasks, a two-layer single-head Transformer achieves an accuracy requiring only O(ε⁻¹) parameters, whereas a single-layer model needs Ω(ε⁻ᵏ) parameters—with k growing linearly with sequence length—to attain the same precision. By integrating attention mechanisms with approximation complexity analysis, the theory not only recovers known bounds but also accurately predicts empirical phenomena, providing an interpretable and quantifiable foundation for the superiority of multilayer Transformers.

Abstract

While the approximation properties of single-layer Transformer architectures have been studied in recent works, a rigorous theoretical understanding of the multi-layer setting remains limited. In this work, we establish that multi-layer Transformers possess fundamentally different approximation capabilities from single-layer ones: for certain retrieval tasks, any single-layer Transformer requires least $Ω(\varepsilon^{-k})$ parameters to achieve precision $\varepsilon$, where $k$ grows linearly with sequence length $T$, whereas a two-layer Transformer with a single head per layer achieves the same approximation precision with at most $O (\varepsilon^{-1})$ parameters. To understand this separation, we identify two structural mechanisms underlying multi-layer approximation. Specifically, softmax attention can only efficiently retrieve the token attaining the maximum attention score, incurring exponential-in-length parameter cost for $k$-th largest retrieval with $k \geq 2$. Moreover, the parameter cost of decoding coupled information scales with the size of the retrieved token set. Motivated by these findings, we propose InfoFlow, a framework for multi-layer Transformers. The framework tracks an information set of accessible input positions at each token and layer, assigning an explicit approximation rate to each mode of information propagation. This abstraction recovers known approximation bounds, remains consistent with experimental observations on trained networks, and yields concrete predictions in settings where direct theoretical analysis is currently intractable. Our results provide a principled framework for reasoning about the approximation efficiency of multi-layer Transformers.

Problem

• multi-layer Transformers
• approximation capability
• parameter efficiency
• information retrieval
• theoretical analysis

Innovation

• multi-layer Transformers
• approximation efficiency
• InfoFlow
• attention mechanism
• parameter complexity