Lifelong In-Context Learning with Transformers Requires Parametric Forms of Attention

2026-06-23

AI Summary

Traditional Transformers struggle to support lifelong continual learning due to the quadratic complexity of their attention mechanism and the ever-growing key-value cache. This work proposes replacing static key-value caches with a parameterized attention mechanism, implemented via online-trainable neural networks, enabling lifelong contextual learning under constant memory consumption. We unify and generalize approaches such as linear attention, state space models, fast weight programmers, and test-time training, systematically analyzing their limitations in memory capacity and online update cost. Our analysis reveals the potential of parameterized attention for building agents with long-horizon reasoning capabilities and identifies key open challenges, thereby establishing a theoretical framework and research directions for future work.

Abstract

Lifelong continual learning remains an obstacle on the path to human-like intelligence. Modern transformers show sparks of intelligence with in-context learning. The quadratic nature of attention, however, prohibits transformers from performing this process on arbitrarily long sequences. In this work, we argue that extending in-context learning to lifelong settings is a practical solution for continual learning in AI agents. In particular, we argue that \emph{parametric forms of attention} are needed to understand a lifetime of context with transformers on a fixed hardware budget. These attention mechanisms learn the relationship between keys and their associated values at test-time with parametric regression. Our generalization of parametric approaches (linear attention, state-space models, fast weight programmers, and test-time training layers) contrasts with nonparametric counterparts like softmax attention. They replace the ever-growing key-value cache with an online-trainable neural network, maintaining a constant memory footprint. We highlight how parametric attention currently fall short of lifelong learning due to limited memory capacity or costly online updates. To address these issues, we pose a set of open questions with novel insights to guide the field toward long-horizon agents.

Problem

• lifelong learning
• in-context learning
• parametric attention
• transformers
• continual learning

Innovation

• parametric attention
• lifelong in-context learning
• transformers
• continual learning
• constant memory footprint