Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks

2026-07-13

AI Summary

This work investigates the mechanisms by which Transformers develop reasoning capabilities in inductive inference tasks. Focusing on a broad class of generalized induction problems, the authors propose a unified theoretical framework that, for the first time, interprets the formation of internal circuits in Transformers as a low-dimensional dynamical system phenomenon. They prove that training dynamics are confined to a low-dimensional invariant manifold. By integrating dynamical systems theory, attention analysis, and synthetic task modeling, the framework characterizes complex learning behaviors through a small set of interpretable coordinates. It elucidates how data statistics govern the competition between in-context and in-weight learning and reveals the critical role of random initialization in determining the final circuit configuration. This approach enables predictive analysis of learning mechanisms and automatic detection of learned circuits.

Abstract

We present a theoretical framework to explain the emergence of inductive reasoning abilities in Transformer language models. While previous works on Transformer learning dynamics have so far been mostly tied to specific tasks, we study a generalized class of inductive tasks that unifies several synthetic tasks known in the literature, including in-context n-grams and multi-hop reasoning. In this class, we theoretically prove that the training dynamics of attention models can be confined to a highly interpretable, low-dimensional invariant manifold. On this manifold, the learning dynamics are captured by a handful of interpretable coordinates rather than millions of parameters, making both theoretical and empirical analysis more tractable. Using this framework, we characterize how data statistics govern the competition between in-context and in-weights learning, we study how random initializations determine the `winning' circuit when multiple solutions are possible, and we demonstrate that the coordinate frame associated with the manifold can be used to automatically detect which circuits have been learned in trained models. By casting circuit formation as a low-dimensional dynamical phenomenon, we take a step toward a predictive theory of how Transformers learn.

Problem

• inductive reasoning
• Transformer
• learning dynamics
• invariant manifold
• circuit formation

Innovation

• invariant manifold
• inductive reasoning
• learning dynamics
• attention mechanisms
• circuit detection