Context Distillation as Latent Memory Management

2026-05-27

AI Summary

This work addresses the challenge that existing context distillation methods struggle to effectively store, retrieve, and safely activate multiple latent memories under non-ideal conditions. Framing this as a latent memory management task, the authors propose a modular memory bank architecture in which a dedicated LoRA adapter is trained for each context. Memory retrieval and activation are precisely controlled through a query-driven routing mechanism. To enhance robustness and prevent interference from irrelevant memories, the method introduces a novel self-gating mechanism that dynamically determines which memories to activate. Additionally, a cache-sharing strategy is devised to reduce inference overhead. Experimental results demonstrate that the proposed approach significantly outperforms current state-of-the-art methods on retrieval-augmented benchmarks, achieving both high efficiency and strong robustness.

Abstract

Context distillation compresses contextual information into model parameters, yet existing methods often ignore how multiple distilled latent memories should be stored, retrieved, and safely activated in non-oracle settings. We formulate context distillation as a latent memory management problem. We distill each context into an independent LoRA adapter, forming a modular memory bank that enables explicit memory selection. Given a query, our framework retrieves candidate memories, routes the query to the most suitable adapter, and uses a Self-Gating mechanism to decide whether latent memory should be activated. To improve efficiency, we further introduce cache sharing to reduce management overhead during inference. Experiments show that our method substantially outperforms baselines with retrieval, while Self-Gating improves robustness by deactivate unnecessary latent memories.

Problem

• context distillation
• latent memory
• memory management
• non-oracle settings
• memory activation

Innovation

• context distillation
• latent memory management
• LoRA adapter
• Self-Gating
• cache sharing