Memory as a Controlled Process: Learned Adaptive Memory Management for LLM Agents

2026-07-15

AI Summary

Existing large language model agents typically rely on static heuristics for memory access, which struggle to adapt to the dynamic demands of tasks. This work addresses this limitation by formulating memory management as a Markov decision process and introducing a lightweight, adaptive memory control framework. The proposed approach requires no pretraining, incurs no additional large language model calls, and is compatible with any backend memory system. It leverages a contextual multi-armed bandit strategy based on Upper Confidence Bound (UCB), learning online from only binary task-level feedback. Extensive experiments across six benchmarks, three agent architectures, and three large language models demonstrate that the method improves average task success rates by up to 15.2% while reducing token consumption by 5%–20%.

Abstract

Large Language Model (LLM) agents increasingly rely on external memory systems to accumulate experience across tasks. Yet nearly all existing approaches, from graph-structured memories to reflective insight stores, access memory through fixed, hand-designed heuristics. We argue that this static view of memory is a core bottleneck for agentic learning because optimal memory behavior is fundamentally context-dependent. The early stages of the tasks, benefit from minimal retrieval because memory is sparse; recurring goal types benefit from plan reuse rather than generic nearest-neighbor lookup; stuck agents benefit from re-retrieval with alternative queries; and across long task streams, the memory store itself must be consolidated and pruned to remain useful. We present Memory as a Controlled Process (MemCon), a framework that models memory operations as a Markov Decision Process and learns an online policy that adaptively decides when, what, and how much to retrieve, when to inject a distilled plan, and when to consolidate or forget. MemCon is backend-agnostic: it wraps any existing memory implementation, learns from task-by-task binary feedback with no pretraining and no additional LLM calls, and uses a lightweight tabular contextual bandit with UCB exploration that converges within tens of tasks. Across 6 benchmarks, 3 agent frameworks, and 3 LLM backbones, MemCon consistently outperforms multiple memory baselines by up to 15.2 points in task success while reducing token consumption by 5--20%.

Problem

• memory management
• LLM agents
• context-dependent memory
• adaptive retrieval
• memory consolidation

Innovation

• adaptive memory management
• Markov Decision Process
• contextual bandit
• LLM agents
• memory consolidation