Scaling Limits of Long-Context Transformers

2026-05-08

AI Summary

This study investigates the degenerate behavior of softmax self-attention in long-context Transformers as the context length tends to infinity. Focusing on the setting where queries are fixed and keys are random, the authors introduce an inverse temperature parameter to systematically characterize a phase transition in attention—from uniform averaging to collapse onto a single key. By integrating tools from probability limit theory, spherical geometry, and random matrix theory, they show that the critical scale governing attention selectivity is determined by the local exponent of the query-key distance distribution. Notably, in the subcritical regime, the attention map approximates the backward heat equation. The work fully establishes the limiting distributions of attention outputs across subcritical, critical, and supercritical regimes, capturing phenomena such as Gaussian fluctuations, finite-neighbor mass retention, and point concentration.

Abstract

We study the long-context limit of softmax self-attention with a fixed query and a random context of $n$ i.i.d. keys on the sphere, viewing the inverse temperature $β_n$ as the scaling parameter that decides whether attention degenerates into uniform averaging or collapses onto the single closest key. We show that the critical scale at which selectivity emerges is determined by the local exponent of the distance-to-query distribution near zero rather than by global features of the context, and scales like $β_n^\ast \asymp n^{2/(d-1)}$ for uniform keys on $\mathbb{S}^{d-1}$. Furthermore, we characterize the limiting laws of the ordered attention weights and of the attention output across all regimes of $β_n$: a subcritical regime in which the output reduces to a local average around $q$ with explicit deterministic bias and Gaussian fluctuations; a critical regime in which a finite collection of nearest keys retains macroscopic mass without single-key collapse; and a supercritical regime in which all mass concentrates on the closest key. Of notable interest is the subcritical case with identity value matrix where the attention map approximately implements a backward heat equation.

Problem

• long-context transformers
• softmax self-attention
• scaling limits
• attention degeneracy
• inverse temperature

Innovation

• long-context transformers
• scaling limits
• softmax attention
• phase transition
• local averaging