Ask Only When Needed: Proactive Retrieval from Memory and Skills for Experience-Driven Lifelong Agents

2026-04-22

AI Summary

Existing lifelong learning agents typically retrieve experiences passively, struggling to proactively identify knowledge gaps and efficiently acquire useful information. This work proposes ProactAgent, a novel framework that, for the first time, formulates retrieval as an explicit policy action. It introduces Proactive RL-based Retrieval (ProactRL) to enable on-demand, active memory access and integrates Experience-Enhanced Online Evolution (ExpOnEvo) to jointly optimize the agent’s policy and a structured experience repository encompassing factual knowledge, episodic memories, and behavioral skills. Evaluated on SciWorld and AlfWorld, the approach achieves success rates of 73.50% and 71.28%, respectively, significantly reducing retrieval overhead while matching the performance of closed-source models on StuLife.

Technology Category

• Search and Optimization: Learning to Search
• Multiagent Systems: Adversarial Agents
• Machine Learning: Active Learning

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Online lifelong learning enables agents to accumulate experience across interactions and continually improve on long-horizon tasks. However, existing methods typically treat retrieval from past experience as a passive operation, triggering it only at task initialization or after completing a step. Consequently, agents often fail to identify knowledge gaps during interaction and proactively retrieve the most useful experience for the current decision. To address this limitation, we present ProactAgent, an experience-driven lifelong learning framework for proactive retrieval over a structured experience base. We first introduce Experience-Enhanced Online Evolution (ExpOnEvo), which enables continual improvement through both policy updates and memory refinement. The experience base organizes historical interactions into typed repositories, including factual memory, episodic memory, and behavioral skills, so that retrieval can provide both relevant evidence and actionable guidance. On top of this, we propose Proactive Reinforcement Learning-based Retrieval (ProactRL), which models retrieval as an explicit policy action and learns when and what to retrieve via paired-branch process rewards. By comparing continuations from identical interaction prefixes with and without retrieval, ProactRL provides step-level supervision for retrieval decisions, encouraging retrieval only when it leads to better task outcomes or higher efficiency. Experiments on SciWorld, AlfWorld, and StuLife show that ProactAgent consistently improves lifelong agent performance, achieving success rates of 73.50\% on SciWorld and 71.28\% on AlfWorld while substantially reducing retrieval overhead, and attains performance competitive with proprietary models on StuLife.

Problem

• lifelong learning
• proactive retrieval
• experience-driven agents
• knowledge gaps
• memory retrieval

Innovation

• proactive retrieval
• lifelong learning
• experience base
• reinforcement learning
• memory organization