FlashMem: Distilling Intrinsic Latent Memory via Computation Reuse

2026-01-09

arXiv.org

AI Summary

This work addresses the limitation of large language models in maintaining long-term autonomy due to the absence of a dynamic contextual memory mechanism, which necessitates redundant processing of historical information. The authors propose an intrinsic memory distillation method that requires no additional encoder, leveraging the final hidden state as a sufficient statistic of interaction history and directly synthesizing memory through a shared key-value (KV) cache. Coupled with a parameter-free cognitive monitor—based on attention entropy—that triggers memory integration only under high cognitive uncertainty, the approach introduces a frozen-cache attention mechanism and a shared KV consolidator. This design achieves performance comparable to heavyweight baselines while reducing inference latency by a factor of five, substantially enhancing both computational efficiency and sustained cognitive capability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

The stateless architecture of Large Language Models inherently lacks the mechanism to preserve dynamic context, compelling agents to redundantly reprocess history to maintain long-horizon autonomy. While latent memory offers a solution, current approaches are hindered by architectural segregation, relying on auxiliary encoders that decouple memory from the reasoning backbone. We propose FlashMem, a framework that distills intrinsic memory directly from transient reasoning states via computation reuse. Leveraging the property that internal representations uniquely encode input trajectories, FlashMem identifies the last hidden state as a sufficient statistic for the interaction history. This enables a Shared-KV Consolidator to synthesize memory by attending directly to the backbone's frozen cache, eliminating redundant re-parameterization. Furthermore, a parameter-free Cognitive Monitor leverages attention entropy to adaptively trigger consolidation only when high epistemic uncertainty is detected. Experiments demonstrate that FlashMem matches the performance of heavy baselines while reducing inference latency by 5 times, effectively bridging the gap between efficiency and persistent cognition.

Problem

• latent memory
• stateless architecture
• computation reuse
• long-horizon autonomy
• dynamic context

Innovation

• computation reuse
• intrinsic latent memory
• Shared-KV Consolidator
• Cognitive Monitor
• attention entropy