Memory Caching: RNNs with Growing Memory

2026-02-27

AI Summary

This work proposes a memory caching mechanism to address the limited fixed memory capacity of recurrent neural networks (RNNs), which significantly hinders their performance on long-range dependency tasks compared to Transformers. By dynamically storing checkpoints of hidden states, the method enables RNNs to scale their effective memory with sequence length. Four cache variants are introduced, integrating gated aggregation and sparse selection strategies to endow RNNs with flexible and efficient memory capabilities for the first time. The approach substantially outperforms existing recurrent models on language modeling, long-context comprehension, and recall tasks, achieving performance close to that of Transformers and effectively narrowing the gap between the two architectures in memory-intensive scenarios.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Transformers have been established as the de-facto backbones for most recent advances in sequence modeling, mainly due to their growing memory capacity that scales with the context length. While plausible for retrieval tasks, it causes quadratic complexity and so has motivated recent studies to explore viable subquadratic recurrent alternatives. Despite showing promising preliminary results in diverse domains, such recurrent architectures underperform Transformers in recall-intensive tasks, often attributed to their fixed-size memory. In this paper, we introduce Memory Caching (MC), a simple yet effective technique that enhances recurrent models by caching checkpoints of their memory states (a.k.a. hidden states). Memory Caching allows the effective memory capacity of RNNs to grow with sequence length, offering a flexible trade-off that interpolates between the fixed memory (i.e., $O(L)$ complexity) of RNNs and the growing memory (i.e., $O(L^2)$ complexity) of Transformers. We propose four variants of MC, including gated aggregation and sparse selective mechanisms, and discuss their implications on both linear and deep memory modules. Our experimental results on language modeling, and long-context understanding tasks show that MC enhances the performance of recurrent models, supporting its effectiveness. The results of in-context recall tasks indicate that while Transformers achieve the best accuracy, our MC variants show competitive performance, close the gap with Transformers, and performs better than state-of-the-art recurrent models.

Problem

• recurrent neural networks
• memory capacity
• sequence modeling
• recall-intensive tasks
• long-context understanding

Innovation

• Memory Caching
• Recurrent Neural Networks
• Growing Memory
• Subquadratic Complexity
• In-Context Recall