Proteus: Incremental Memory Activation for Long-Context Sequence Modeling

2026-08-17

AI Summary

This study addresses early contamination and late-stage capacity bottlenecks caused by static memory in long-context modeling by proposing an Incremental Memory Activation paradigm. This approach progressively expands effective capacity as sequence length increases, employing early compression to distill historical information while dynamically unlocking new space, thereby optimizing capacity scheduling in general-purpose memory architectures with zero additional overhead. The mechanism integrates seamlessly into state-of-the-art models such as SWLA and Titans, yielding consistent improvements across language modeling, reasoning, and long-document retrieval tasks. Notably, performance gains scale significantly with context length, effectively overcoming the efficiency ceiling associated with long-range dependencies.

Abstract

The quadratic cost of attention-based sequence models for long contexts has motivated a growing line of research on memory-based models that can compress context into a compact state. However, most existing memory models expose a static memory throughout the entire sequence. Because early tokens face no compression pressure, they occupy too many degrees of freedom and "pollute" the memory state, leaving little capacity for later context and increasing interference between what is stored and what arrives next. We study a new paradigm of incremental memory activation, where the effective capacity of memory is progressively expanded as the context grows. Imposing an early bottleneck forces the model to compress history more effectively, while unlocking fresh capacity over time reduces interference and improves retention of later context. We instantiate this paradigm in Proteus, a straightforward mechanism that can be incorporated into a broad class of neural memory architectures at no additional cost. We apply Proteus to state-of-the-art models, including SWLA, Comba, Titans, and Hope-Attention, and observe consistent improvements on standard language modeling and reasoning, as well as on long-context retrieval and understanding, with gains that grow at longer context lengths. Overall, our results show that static memory is suboptimal and that scheduling effective capacity is a simple and broadly applicable tool for sequence modeling.

Problem

• Long-Context Sequence Modeling
• Memory-based Models
• Static Memory
• Memory Interference
• Context Compression

Innovation

• Incremental Memory Activation
• Long-Context Sequence Modeling
• Memory Compression
• Dynamic Capacity Scheduling
• Proteus