Fast weight programming and linear transformers: from machine learning to neurobiology

2025-08-11

AI Summary

Conventional RNNs, with vector-valued hidden states, struggle to model dynamic synaptic weight adjustments underlying short-term memory. Method: We propose a Fast Weight Programmer (FWP) recurrent architecture featuring *matrix-valued* hidden states, where the hidden state itself is a learnable weight matrix updated in real time via a programmer network using gradient descent—enabling efficient short-term memory storage. Contribution/Results: We establish that FWP is theoretically equivalent to linear attention and constitutes a recurrent realization of the Transformer; it also exhibits formal unification with discrete-time state space models (SSMs). Crucially, its online weight update mechanism closely mirrors Hebbian synaptic plasticity. This work is the first to systematically bridge FWP across three domains: machine learning architecture design, sequence modeling theory, and neurobiological mechanisms—thereby offering a novel unifying paradigm for intelligent systems.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Deep Neural Architectures and Foundation Models
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Recent advances in artificial neural networks for machine learning, and language modeling in particular, have established a family of recurrent neural network (RNN) architectures that, unlike conventional RNNs with vector-form hidden states, use two-dimensional (2D) matrix-form hidden states. Such 2D-state RNNs, known as Fast Weight Programmers (FWPs), can be interpreted as a neural network whose synaptic weights (called fast weights) dynamically change over time as a function of input observations, and serve as short-term memory storage; corresponding synaptic weight modifications are controlled or programmed by another network (the programmer) whose parameters are trained (e.g., by gradient descent). In this Primer, we review the technical foundations of FWPs, their computational characteristics, and their connections to transformers and state space models. We also discuss connections between FWPs and models of synaptic plasticity in the brain, suggesting a convergence of natural and artificial intelligence.

Problem

• Study dynamic synaptic weight changes in neural networks
• Explore connections between FWPs and brain plasticity
• Review computational traits of Fast Weight Programmers

Innovation

• 2D matrix-form hidden states
• Fast Weight Programmers (FWPs)
• Dynamic synaptic weight modifications