Rethinking Experience Utilization in Self-Evolving Language Model Agents

2026-05-07

AI Summary

Existing self-evolving language model agents lack dynamic judgment in deciding when and how to utilize past experiences, often resorting to fixed injection strategies. This work proposes ExpWeaver, the first systematic approach to address the timing of experience utilization by treating experiences as on-demand resources during reasoning rather than mandatory inputs. ExpWeaver employs a lightweight runtime mechanism that integrates causal ablation, entropy analysis, and reinforcement learning to enable selective experience retrieval. Extensive experiments across four frameworks, seven large language models, and three task categories demonstrate that ExpWeaver consistently achieves state-of-the-art performance, validating the effectiveness and generalizability of dynamic experience utilization. This study shifts the research focus from “what experiences to store” to “how and when to use them.”

Abstract

Self-evolving agents improve by accumulating and reusing experience from past interactions. Existing work has largely focused on how experience is constructed, represented, and updated, while paying less attention to how experience should be used during runtime decision-making. As a result, most agents rely on rigid usage strategies, either injecting experience once at initialization or at every step, without considering whether it is needed for the current decision. This paper studies experience utilization as a critical design dimension of self-evolving agents. We ask whether agents benefit from interweaving experience use with decision-making, so that experience is invoked only when additional guidance is needed. To examine this question, we introduce {ExpWeaver}, a lightweight instantiation that leaves experience construction unchanged and modifies only runtime utilization by exposing experience as an optional resource during reasoning. Across four representative frameworks, seven LLM backbones, and three types of environments, ExpWeaver consistently achieves the best performance among different utilization strategies. Reinforcement learning experiments further show that this behavior can be amplified through training. Usage-pattern, causal ablation, and entropy-based analyses reveal that ExpWeaver enables agents to invoke experience selectively, at beneficial decision points, and under higher reasoning uncertainty. Overall, our findings call for a shift from merely studying \emph{what} experience to store toward understanding \emph{how} and \emph{when} experience should enter decision-making.

Problem

• experience utilization
• self-evolving agents
• runtime decision-making
• language model agents
• adaptive experience use

Innovation

• experience utilization
• self-evolving agents
• selective reasoning
• ExpWeaver
• runtime decision-making