A Hippocampus for Linear Attention: An Exact Memory for What the Recurrent State Forgets

2026-07-02

AI Summary

This work addresses the limitation of linear attention mechanisms, whose fixed-size recurrent states tend to overwrite early critical information, impairing precise recall. Inspired by complementary learning systems, the authors propose the HOLA architecture, which preserves a compressed state while introducing a bounded key-value cache to decouple structural modeling from essential memory storage. The approach eliminates the need for learned eviction policies by updating the compressed state via a delta rule and selectively writing to the cache based on prediction residuals (β‖e‖). Accurate retrieval is further enabled through a decoupled RMSNorm-γ mechanism. Evaluated on Wikitext, a 340M-parameter HOLA model achieves a perplexity of 22.92—a 16.1% improvement over Transformer++ with full attention—and demonstrates superior performance on 32k-length needle-in-a-haystack tasks.

Abstract

Linear-attention and state-space language models compress the prefix into a fixed-size recurrent state, yielding O(1) memory at the cost of a lossy exact memory: when many key--value associations compete, earlier facts are overwritten and needle recall degrades. Inspired by Complementary Learning Systems, we give linear attention a hippocampal complement. HOLA (Hippocampal Linear Attention) keeps the usual delta-rule state as a compressive memory and adds a bounded exact KV cache, forming a semiparametric test-time memory: the state models linearly compressible structure, while the cache stores associations that should not be forced through that state. The cache writes without a learned eviction module, keeping tokens with large beta * ||e||, the prediction residual actually committed to the state; a decoupled RMSNorm-gamma cache read then turns these exact KV pairs into sharp retrieval rather than soft averaging. At 340M parameters trained on 15B SlimPajama tokens, HOLA lowers Wikitext perplexity from 27.32 to 22.92 (-16.1%), below a full-attention Transformer++ (26.88), and improves LAMBADA perplexity from 30.95 to 30.26. It also achieves the best linear in-context retrieval and remains much more robust than GDN or a matched HOLA+recency cache on RULER needle-in-a-haystack recall out to 32k tokens (16x its training length).

Problem

• linear attention
• exact memory
• needle-in-a-haystack recall
• recurrent state
• memory compression

Innovation

• linear attention
• hippocampal memory
• exact KV cache
• complementary learning systems
• in-context retrieval