MemDefrag: Latent Memory Defragmentation for Large Language Models

2026-07-07

AI Summary

This work addresses the significant performance degradation of large language models during long-term memory updates, often caused by positional encoding misalignment and the absence of effective target-tracking mechanisms. The authors propose a training-free, model-agnostic memory management framework that leverages attention density from intermediate Transformer layers as an intrinsic tracking signal to dynamically rank, filter, and reorder memories. When memory capacity is exceeded, the framework employs an information-guided proportional forgetting strategy. This approach substantially enhances the robustness of memory updating, achieving a knowledge retention rate of 43.0% after 50 update cycles—more than double that of MemoryLLM and M+ (17.4% and 17.6%, respectively)—and demonstrates strong generalization capabilities in long-context tasks.

Abstract

Latent memory, which stores past knowledge fragments as per-layer hidden states, has emerged as a promising paradigm (e.g., MemoryLLM and M+) for long-term memory in large language models (LLMs). However, the paradigm suffers from significant performance degradation during memory updates, due to positional encoding misalignment and the absence of any tracing mechanism to distinguish target memory fragments from irrelevant ones. To discover such a tracing mechanism, we probe the layer-wise attention density over stored memory fragments, and find that a small set of middle transformer layers consistently concentrates the highest density on the target fragment - exposing an inherent tracing signal. In light of this, we propose MemDefrag, a training-free and model-agnostic framework that (1) uses a middle-layer tracing signal to conduct memory defragmentation (rank, reorder, and filter memories), and (2) applies an informativeness-guided proportional forgetting mechanism once capacity is exceeded. Experiments show that MemDefrag substantially outperforms MemoryLLM and M+ on knowledge retention (e.g., 43.0% vs. 17.4%/17.6% after 50 memory updates) and long-context benchmarks, and generalizes well across various LLMs and latent-memory variants.

Problem

• latent memory
• memory defragmentation
• large language models
• memory update
• attention density

Innovation

• latent memory
• memory defragmentation
• attention density
• proportional forgetting
• model-agnostic