Analogies between Transformer Layers and Power Method

2026-05-25

AI Summary

This work uncovers a deep connection between the internal operations of Transformer layers and the classical numerical algorithm known as the power method. By interpreting the projections in self-attention together with layer normalization as a single iteration of the power method, the authors theoretically demonstrate that token representations progressively align with the dominant eigenvector of the product of the value matrix and the output weight matrix during forward propagation. This study establishes the first rigorous analogy between Transformers and the power method, introducing a novel paradigm for steering model outputs by manipulating the direction of this dominant eigenvector. Leveraging linear algebraic analysis and a shared-weight architecture, the work both analytically characterizes and empirically validates this alignment phenomenon, demonstrating the feasibility of targeted interventions to control model behavior.

Abstract

In the paper we show that there is an analogy between the operations occurring in a layer of a transformer (projections and layer normalizations, disregarding the feedforward neural network) and a step in the power method. Coherently with this analogy, we show that passing through a layer the tokens tend to be tilted towards the principal eigenvector of a matrix which is the product of the output and value weight matrices of that layer. In the special case of a transformer with shared weights (i.e., in which all layers have identical weights) then the alignment with this principal eigenvector is particularly evident empirically, and can also be shown analytically. The analogy also suggests a method to steer the output of the transformer towards an arbitrary desired direction in token space.

Problem

• Transformer
• Power Method
• Eigenvector
• Layer Normalization
• Weight Matrices

Innovation

• Transformer
• Power Method
• Principal Eigenvector
• Token Alignment
• Shared Weights