Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity

2026-07-08

AI Summary

This work addresses the limited state capacity of linear attention models, which hinders their performance on long-context recall tasks, while expanding the state incurs prohibitive computational costs. To overcome this limitation, the authors propose the Sparse Delta Memory (SDM) architecture, which extends Gated DeltaNet by incorporating a large-scale explicit memory bank. SDM replaces the conventional dense key-value outer-product operations with a sparse read-write mechanism and employs learnable initial states as parameterized memory. Without increasing computational or parameter overhead, this approach substantially enhances model performance on in-context learning, long-context retrieval, and commonsense reasoning tasks.

Abstract

Linear attention models allow a fixed state size and a fixed amount of compute per token. However, due to their limited state size, linear attention models fall behind in long-context recall compared to softmax-attention-based transformer architectures. Increasing the state size of linear attention improves recall performance but at the cost of higher FLOPs. In this work, we introduce Sparse Delta Memory (SDM), an architecture that scales the hidden state of gated linear RNNs to orders of magnitude higher capacity using a sparse addressing scheme. SDM extends the Gated DeltaNet architecture by replacing the dense key-value outer product with sparse reads and writes to a large explicit memory. We show that, under an isoFLOP constraint and with an identical number of parameters, a higher state memory capacity significantly improves performance on in-context learning and long-context retrieval tasks. Moreover, by learning the initial state of the SDM memory and therefore using it as a parametric memory, we show that the model further improves on a wide range of common-knowledge and reasoning tasks.

Problem

• linear RNNs
• state capacity
• long-context recall
• sparse memory
• in-context learning

Innovation

• Sparse Delta Memory
• linear RNNs
• sparse addressing
• in-context learning
• long-context retrieval