The Recurrent Transformer: Greater Effective Depth and Efficient Decoding

2026-04-22

AI Summary

This work addresses the limitations of conventional Transformers, which are constrained by finite effective depth, and recurrent models, which suffer from optimization instability and poor hardware efficiency despite their theoretically infinite depth. The authors propose a recurrent Transformer architecture that incorporates intra-layer recurrent attention, wherein each layer dynamically computes key-value pairs based on its own historical activations. This design significantly increases effective model depth while preserving the computational cost of standard autoregressive decoding. By unifying Transformer and token-level recurrence, the approach avoids training instability and achieves superior performance with fewer layers. Furthermore, a block-wise exact algorithm reduces HBM traffic from Θ(N²) to Θ(N log N), yielding an arithmetic intensity of Θ(N / log N). Experiments demonstrate that the model outperforms standard Transformer baselines of comparable size on C4 pretraining tasks with 150M and 300M parameters.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Transformers process tokens in parallel but are temporally shallow: at position $t$, each layer attends to key-value pairs computed based on the previous layer, yielding a depth capped by the number of layers. Recurrent models offer unbounded temporal depth but suffer from optimization instability and historically underutilize modern accelerators. We introduce the Recurrent Transformer, a simple architectural change where each layer attends to key-value pairs computed off its own activations, yielding layerwise recurrent memory while preserving standard autoregressive decoding cost. We show that the architecture can emulate both (i) a conventional Transformer and (ii) token-to-token recurrent updates under mild assumptions, while avoiding optimization instability. Naively, prefill/training appears bandwidth-bound with effective arithmetic intensity near $1$ because keys and values are revealed sequentially; we give an exact tiling-based algorithm that preserves the mathematical computation while reducing HBM traffic from $Θ(N^2)$ to $Θ(N\log N)$, increasing effective arithmetic intensity to $Θ(N/\log N)$ for sequence length $N$. On 150M and 300M parameter C4 pretraining, Recurrent Transformers improve cross-entropy over a parameter-matched Transformer baseline and achieve the improvement with fewer layers (fixed parameters), suggesting that recurrence can trade depth for width, thus reducing KV cache memory footprint and inference latency.

Problem

• Recurrent Transformer
• temporal depth
• optimization instability
• hardware efficiency
• autoregressive decoding

Innovation

• Recurrent Transformer
• effective depth
• tiling algorithm
• KV cache reduction
• arithmetic intensity