Context by Distinct Information: An Auditable Dirichlet-Process Working Memory for Long, Redundant Context Streams

2026-07-11

AI Summary

This work addresses the inefficiency of traditional token-level context modeling in distinguishing between recollective, summarizing, and local information. The authors propose a novelty-driven memory mechanism that dynamically partitions context into three components: a content-addressable novelty cache for retrievable details, a recurrent state for compressed summaries, and a sliding window for recent local context. This architecture uniquely scales memory capacity with the amount of distinct information rather than raw token count, yielding an auditable and interpretable working memory structure. Integrating a Dirichlet-process-inspired novelty-gated attention mechanism, the system achieves full-attention performance in character-level control tasks with roughly half the attention cost and outperforms both full-attention and fixed-budget baselines on a thousand-event healthcare claims prediction task, while enabling human inspection of stored memory contents.

Abstract

Context engineering decides what information a model carries forward, and current designs meter it in tokens: compressing the past into a bounded recurrent state, keeping a key-value entry for every token, or imposing a fixed budget through a window or eviction rule. All three make the token the unit of memory even when the stream is redundant and the task depends on the distinct information it carries. Building on a companion mechanism paper that opens a cache slot only when an incoming key is novel, so memory scales with the number of distinct items rather than tokens, we develop that allocate-on-novelty cache as a working-memory component and organize context by how a task depends on the past: recall-carried information belongs in a content-addressed novelty cache, summary-carried information in a recurrent state, and locality-carried information in a recency window. The claim is empirical and bounded. On a matched character-level control, novelty-gated attention reaches full-attention performance while attending to about half the tokens, and coupling the cache with a state-space summary matches full-attention coupling at that reduced cost; the advantage grows as context lengthens, while a sliding window is preferable on short, locality-dominated spans. On next-code prediction over synthetic Medicare claims the coupled component leads full attention and every fixed-budget eviction policy at a thousand-event horizon, whereas cost forecasting over the same stream is summary-carried and the cache is neutral. The retained memory is an inspectable table of templates, codes, drugs, or places rather than an opaque state. The experiments are small-scale and use only public data; they establish the primitive that context can scale with distinct information rather than tokens, in a working memory that is content-addressable and auditable.

Problem

• context engineering
• working memory
• distinct information
• redundant context
• auditable memory

Innovation

• novelty-gated attention
• Dirichlet-process memory
• content-addressable cache
• auditable working memory
• distinct information scaling