From Self-Attention to Connection Laplacian: A Unified Operator View of Transformers

2026-07-12

AI Summary

This work addresses the lack of a unified geometric understanding of self-attention mechanisms at the operator level. By modeling token sequences as vector fields on graphs, the study rigorously formalizes single-head and multi-head self-attention as connection propagation processes for the first time, establishing their equivalence to random-walk connection Laplacians under precise conditions. Building on connection-walk theory, the authors develop an operator-level geometric analysis framework. Empirical results demonstrate that attention graphs in Transformers of varying scales and architectures converge in deep layers to stable geometric operators, whose transport maps self-organize into near-scaled isometries—a phenomenon that intensifies with model scale, revealing intrinsic geometric regularities underlying self-attention.

Abstract

Self-attention is a ubiquitous primitive in modern sequence models, yet its operator-level geometry is only partially understood. We view a token sequence as a vector field over the token-position graph and identify attention as a connection walk: messages are aggregated by a nonnegative walk matrix while being transported along each edge by a learned linear map. Within this framework, we prove that single-head attention (SHA) is exactly a connection propagation step with constant transport, and that multi-head attention (MHA) is exactly a single edge-dependent connection walk whose effective transport is an attention-gated mixture of headwise transports. We further clarify the conditions under which the corresponding generator reduces to a random-walk connection Laplacian, highlighting the roles of stochasticity, reversibility, and metric-compatible transports. Empirically, we find that trained Transformers across scales (from 124M to 8B) and structures (encoder/decoder) exhibit geometric structure consistent with our theory: effective attention graphs converge to stable geometric operators in deeper layers, learned transports self-organize into approximate scaled isometries, and both phenomena strengthen consistently with scale. Overall, the paper provides a precise connection-walk formalism that links self-attention to classical geometric operators, along with a set of operator-level tools for analyzing transformer models from a geometric perspective.

Problem

• self-attention
• connection Laplacian
• transformers
• geometric operators
• operator-level geometry

Innovation

• connection Laplacian
• self-attention
• geometric operator
• vector field
• transformer geometry