When Do Transformers Outperform Feedforward and Recurrent Networks? A Statistical Perspective

2025-03-14

AI Summary

This work investigates the sample complexity advantage of Transformers over feedforward networks (FFNs) and recurrent neural networks (RNNs) for dynamic sparse sequence modeling—tasks where output depends only on a small, unknown subset of input tokens. Method: We formalize a sparse retrieval-and-generation task and conduct a statistical learning–theoretic analysis, combining attention mechanism modeling with rigorous lower-bound proofs. Contribution/Results: We establish the first strict sample complexity hierarchy among these architectures for this task: (i) a single-layer Transformer with at least $ q $ attention heads achieves $ O(1) $ sample complexity—constant in sequence length $ N $; (ii) RNNs require $ N^{Omega(1)} $ samples; and (iii) FFNs still require $ Omega(N) $ samples. Our analysis reveals that Transformers attain fundamental statistical efficiency gains via adaptive sparse attention, providing theoretical justification for their architectural superiority in sparse dependency modeling.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Natural Language Processing: Learning & Optimization for NLP
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Theoretical efforts to prove advantages of Transformers in comparison with classical architectures such as feedforward and recurrent neural networks have mostly focused on representational power. In this work, we take an alternative perspective and prove that even with infinite compute, feedforward and recurrent networks may suffer from larger sample complexity compared to Transformers, as the latter can adapt to a form of dynamic sparsity. Specifically, we consider a sequence-to-sequence data generating model on sequences of length $N$, in which the output at each position depends only on $q$ relevant tokens with $q ll N$, and the positions of these tokens are described in the input prompt. We prove that a single-layer Transformer can learn this model if and only if its number of attention heads is at least $q$, in which case it achieves a sample complexity almost independent of $N$, while recurrent networks require $N^{Omega(1)}$ samples on the same problem. If we simplify this model, recurrent networks may achieve a complexity almost independent of $N$, while feedforward networks still require $N$ samples. Consequently, our proposed sparse retrieval model illustrates a natural hierarchy in sample complexity across these architectures.

Problem

• Transformers outperform feedforward and recurrent networks in sample complexity.
• Transformers adapt to dynamic sparsity, reducing sample complexity for sequence tasks.
• Recurrent networks require more samples than Transformers for sparse sequence models.

Innovation

• Transformers adapt to dynamic sparsity efficiently
• Single-layer Transformers require attention heads ≥ q
• Transformers achieve sample complexity independent of N