Adaptive inference and function vectors in deep transformers

2026-06-15

AI Summary

This work investigates how deep Transformers achieve adaptive in-context reasoning under constraints of communication, locality, and depth. By modeling the network as a mean-field interacting system, the authors introduce “function vectors” to represent internal states and develop a distributed inference theory that reveals a nontrivial relationship between network depth and the hierarchical structure of latent variables. Within this theoretical framework, they integrate a constrained linear attention architecture and validate its predictions on in-context regression tasks, demonstrating that deep Transformers possess expressive adaptive reasoning capabilities surpassing those of prior models.

Abstract

Transformers are widely used as a general-purpose substrate for learning complex correlations between a large collection of coupled variables, but their internal mechanisms have remained mysterious. We introduce a theory of a deep transformer as a mean-field interacting system that implements distributed inference, subject to constraints on communication, locality and depth. We show that such a system can exploit internal state representations ('function vectors') to infer a latent context variable at increasingly finer scales over its layers. In an in-context regression task, the theory predicts a non-trivial relationship between non-Gaussian, hierarchical structure in the latent context variable, and transformer depth. Predictions are tested using constrained linear attention transformers and demonstrate adaptive inference in deep architectures. Feedforward blocks and depth enable transformers to implement a much richer class of in-context learning algorithms than previously described.

Problem

• adaptive inference
• function vectors
• deep transformers
• latent context variable
• in-context learning

Innovation

• adaptive inference
• function vectors
• deep transformers
• in-context learning
• mean-field theory