ProcMEM: Learning Reusable Procedural Memory from Experience via Non-Parametric PPO for LLM Agents

2026-02-02

AI Summary

Large language model (LLM) agents often struggle to reuse past experiences in recurring scenarios, leading to computational redundancy and behavioral instability. To address this, this work proposes the Skill-MDP framework, which converts interaction histories into executable skills and introduces a non-parametric variant of Proximal Policy Optimization (PPO). By integrating semantic gradient-guided skill generation with a PPO-based gating mechanism, the approach enables procedural memory learning and validation without requiring parameter updates. Coupled with a score-driven memory maintenance strategy, the method significantly enhances skill reuse and task performance across in-domain, cross-task, and cross-agent settings, while achieving extremely high compression ratios in storing procedural memories.

Technology Category

• Multiagent Systems: Multiagent Learning
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

LLM-driven agents demonstrate strong performance in sequential decision-making but often rely on on-the-fly reasoning, re-deriving solutions even in recurring scenarios. This insufficient experience reuse leads to computational redundancy and execution instability. To bridge this gap, we propose ProcMEM, a framework that enables agents to autonomously learn procedural memory from interaction experiences without parameter updates. By formalizing a Skill-MDP, ProcMEM transforms passive episodic narratives into executable Skills defined by activation, execution, and termination conditions to ensure executability. To achieve reliable reusability without capability degradation, we introduce Non-Parametric PPO, which leverages semantic gradients for high-quality candidate generation and a PPO Gate for robust Skill verification. Through score-based maintenance, ProcMEM sustains compact, high-quality procedural memory. Experimental results across in-domain, cross-task, and cross-agent scenarios demonstrate that ProcMEM achieves superior reuse rates and significant performance gains with extreme memory compression. Visualized evolutionary trajectories and Skill distributions further reveal how ProcMEM transparently accumulates, refines, and reuses procedural knowledge to facilitate long-term autonomy.

Problem

• procedural memory
• experience reuse
• LLM agents
• computational redundancy
• execution instability

Innovation

• Procedural Memory
• Non-Parametric PPO
• Skill-MDP
• LLM Agents
• Experience Reuse