Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning

2025-08-27

AI Summary

Large language models (LLMs) suffer from statelessness and limited context windows, hindering long-horizon reasoning. Existing external memory approaches rely heavily on static, hand-crafted heuristics and lack dynamic, content-aware memory control. To address this, we propose the first end-to-end learnable memory management framework grounded in reinforcement learning (RL). Our method introduces a dual-agent architecture comprising a Memory Manager and a Response Agent, jointly optimizing structured memory operations—ADD, UPDATE, and DELETE—via policy learning. We employ outcome-oriented RL algorithms, including PPO and GRPO, enabling effective fine-tuning under minimal supervision (only 152 question-answer pairs). Experiments demonstrate substantial improvements over state-of-the-art baselines across diverse complex reasoning tasks—including multi-step deduction, temporal reasoning, and knowledge-intensive QA—while exhibiting strong generalization and robustness across different LLM backbones.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Multiagent Learning
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models (LLMs) have demonstrated impressive capabilities across a wide range of NLP tasks, but they remain fundamentally stateless, constrained by limited context windows that hinder long-horizon reasoning. Recent efforts to address this limitation often augment LLMs with an external memory bank, yet most existing pipelines are static and heuristic-driven, lacking any learned mechanism for deciding what to store, update, or retrieve. We present Memory-R1, a reinforcement learning (RL) framework that equips LLMs with the ability to actively manage and utilize external memory through two specialized agents: a Memory Manager that learns to perform structured memory operations {ADD, UPDATE, DELETE, NOOP}, and an Answer Agent that selects the most relevant entries and reasons over them to produce an answer. Both agents are fine-tuned with outcome-driven RL (PPO and GRPO), enabling adaptive memory management and use with minimal supervision. With as few as 152 question-answer pairs and a corresponding temporal memory bank for training, Memory-R1 outperforms the most competitive existing baseline and demonstrates strong generalization across diverse question types and LLM backbones. Beyond presenting an effective approach, this work provides insights into how RL can unlock more agentic, memory-aware behaviors in LLMs, pointing toward richer, more persistent reasoning systems.

Problem

• Enhancing LLMs to actively manage external memory operations
• Overcoming static heuristic-driven memory pipelines with reinforcement learning
• Enabling adaptive memory utilization for long-horizon reasoning tasks

Innovation

• Reinforcement learning framework for memory management
• Two specialized agents for memory operations and reasoning
• Outcome-driven RL fine-tuning with minimal supervision