Dynamic Compression in Recurrent Networks

2026-08-18

AI Summary

研究通过引入动态压缩方法，让循环模型选择性地重新访问过去的信息并更新状态，从而更有效地使用固定大小的状态处理长上下文。

Abstract

Recurrent models process long contexts efficiently by compressing their history into a fixed-size state, but modern architectures typically do so in a single causal pass over the sequence. Each input must therefore be compressed before the model knows how it will later be used, forcing a limited state to compromise across possible future demands. We introduce dynamic compression, which allows a recurrent model to selectively revisit past tokens and revise its fixed-size state through additional recurrent updates. The model need not preserve every part of the history at uniformly high fidelity in its recurrent state, because lower-fidelity information can be revisited from the retained raw sequence when it becomes relevant. We study this in a controlled setting where the model first learns multiple functions in-context and, later in the same sequence, encounters a series of few-shot tasks that each require it to identify and reuse one of those functions. A single-pass model must preserve every function at sufficient fidelity for any future task, whereas selective re-scanning allows the model to revisit and refine only the function currently needed. We find that dynamic compression substantially reduces the recurrent state required for accurate reuse and scales more favorably as the number of stored functions grows. These results demonstrate a computation--memory tradeoff in which recurrent models can spend more computation revisiting their history to make more effective use of a fixed-size state.

Problem

• Recurrent Networks
• Dynamic Compression
• Fixed-size State
• Long Sequences
• Causal Pass

Innovation

• dynamic compression
• recurrent networks
• revisit past tokens
• fixed-size state
• computation--memory tradeoff