Meta-Cognitive Memory Policy Optimization for Long-Horizon LLM Agents

2026-05-28

AI Summary

This work addresses the degradation of reasoning performance in existing memory-augmented large language models during long-horizon tasks, which stems from recursive summarization that discards task-relevant information and accumulates semantic noise, thereby inducing belief bias. To mitigate this issue, the authors propose the Metacognitive Memory Policy Optimization (MMPO) framework, which introduces belief entropy as a self-supervised proxy metric to enable fine-grained supervision over the clarity of intermediate memory states, allowing dynamic optimization of memory strategies. By integrating self-supervised learning, reinforcement learning, and recursive memory summarization, MMPO substantially enhances long-horizon reasoning capabilities, maintaining 97.1% of its performance even within contexts spanning 1.75 million tokens—significantly outperforming current state-of-the-art approaches.

Abstract

Memory-augmented LLM agents tackle complex long-horizon tasks by recursively summarizing interaction trajectories into compact memory. However, existing approaches typically train these memory policies using outcome-based reinforcement learning, failing to localize where intermediate memory quality degrades. As interactions unfold, ambiguous recursive summaries progressively discard task-relevant information and introduce semantic noise. This exacerbates belief deviation, obscuring the agent's estimate of the latent task state and ultimately derailing long-horizon reasoning. We therefore argue that memory optimization should focus not merely on trajectory-level success, but on the clarity of the belief induced by intermediate summaries. To this end, we introduce Belief Entropy, a self-supervised proxy that probes how uncertain the model remains about the latent task state given its current memory. Based on this proxy, we propose Metacognitive Memory Policy Optimization (MMPO). Instead of relying only on sparse outcome-based signals, MMPO provides fine-grained, memory-specific supervision via explicitly penalizing summaries that induce high epistemic uncertainty. Experiments show that MMPO consistently outperforms existing methods on diverse long-horizon tasks, maintaining 97.1% performance even when scaled to 1.75M-token contexts.

Problem

• long-horizon reasoning
• memory-augmented LLM agents
• belief deviation
• recursive summarization
• semantic noise

Innovation

• Metacognitive Memory Policy Optimization
• Belief Entropy
• Memory-Augmented LLM Agents
• Long-Horizon Reasoning
• Self-Supervised Memory Optimization