When Continual Learning Moves to Memory: A Study of Experience Reuse in LLM Agents

2026-04-29

AI Summary

This work addresses the stability–plasticity dilemma in continual learning for large language model agents, which manifests as competition between old and new experiences during external memory retrieval. The authors propose a (k,v) memory framework that decouples experience representation from organization, enabling systematic investigation of memory reuse mechanisms in ALFWorld and BabyAI. Their findings reveal that external memory does not eliminate continual learning challenges but reframes them as issues of representation and retrieval design. Specifically, abstract procedural memories outperform detailed trajectories, while fine-grained memory organization exacerbates forgetting. Abstract memories also facilitate better transfer, though negative transfer disproportionately affects more difficult tasks. Notably, certain design choices that enhance forward transfer simultaneously induce severe catastrophic forgetting.

Abstract

Memory-augmented LLM agents offer an appealing shortcut to continual learning: rather than updating model parameters, they accumulate experience in external memory, seemingly sidestepping the stability-plasticity dilemma of parametric learning. We show that this challenge does not disappear but resurfaces at the memory level. Under a limited context window, old and new experiences compete during retrieval, relocating the continual-learning bottleneck from parameter updates to memory access. To study this phenomenon, we introduce a (k,v) framework that disentangles two fundamental design axes of external memory: how experience is represented and how it is organized for retrieval. Across sequential-task experiments in ALFWorld and BabyAI, we find that abstract procedural memories transfer more reliably than detailed trajectories, while negative transfer disproportionately harms the hard cases. Moreover, finer-grained memory organization is not universally beneficial: designs that yield strong forward transfer can simultaneously induce severe forgetting. Together, these results reveal that external memory does not resolve the continual-learning problem; it reshapes it into a problem of memory representation and retrieval design.

Problem

• continual learning
• memory-augmented LLM agents
• experience reuse
• memory retrieval
• stability-plasticity dilemma

Innovation

• continual learning
• memory-augmented LLM agents
• experience reuse
• memory representation
• retrieval design