Context-Gated Associative Retrieval: From Theory to Transformers

2026-05-08

AI Summary

Traditional associative memory models treat retrieval as a static mapping, neglecting the dynamic modulation of recall by external context. This work proposes a two-stage context-gated associative memory architecture that dynamically reshapes the energy landscape before and after retrieval via gated subcircuits, enabling context-guided memory access. The mechanism introduces context gating for the first time, with theoretical guarantees of enhanced memory separability and sparsity, and reveals a unique self-consistent fixed point driven jointly by direct bias and second-order feedback. Integrating generalized Hopfield networks, statistical physics models, and Transformer architectures, we validate its first-order approximation on Llama-3, demonstrating that in-context learning is equivalent to context-gated retrieval—where native dynamics precisely locate memory subspaces, support zero-shot queries, and yield theoretically predicted exponential gains in retrieval performance.

Abstract

Hopfield networks and their generalizations have established deep connections among biological associative memories, statistical physics, and transformers. Yet most models treat retrieval as a fixed query-to-memory mapping, ignoring the role of external context in recall. In this work, we propose a two-stage associative memory architecture, wherein a context-gate subcircuit reshapes the retrieval energy landscape before and during recall. We show theoretically that context gating increases inter-memory separation while inducing sparsity, translating into exponential improvements in retrieval. Crucially, we prove that the system admits a unique self-consistent fixed point, revealing that the resulting retrieval state is driven by both a direct contextual bias and a second-order retrieval-gate feedback loop. We then bridge this theory to transformers; specifically, we evaluate a first-order approximation on Llama-3, confirming that in-context learning acts as context-gated retrieval. Native dynamics mirror our theory: context localizes a memory subspace, enabling the zero-shot query to cleanly discriminate. Ultimately, this framework provides a mechanistic link between associative memory theory and LLM phenomenology.

Problem

• associative memory
• context gating
• retrieval
• transformers
• in-context learning

Innovation

• context-gated retrieval
• associative memory
• Hopfield networks
• transformers
• in-context learning