Experience Replay Addresses Loss of Plasticity in Continual Learning

2025-03-25

AI Summary

In continual learning, deep neural networks suffer from “plasticity loss”—a severe degradation in adaptability to new tasks—after multi-task backpropagation training. This work demonstrates that memory-based experience replay fundamentally mitigates this phenomenon and, for the first time, shows that replay alone—when integrated with standard Transformer architectures—fully restores continual plasticity. Crucially, no modifications to backpropagation, activation functions, or regularization are required; instead, replay triggers the Transformer’s inherent in-context learning capability. Evaluated across diverse continual learning benchmarks—including regression, classification, and policy evaluation—the method eliminates catastrophic forgetting entirely, sustaining long-term performance at the level of the initial network. These results establish memory-based replay coupled with off-the-shelf Transformers as an effective, non-invasive paradigm for continual adaptation.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Loss of plasticity is one of the main challenges in continual learning with deep neural networks, where neural networks trained via backpropagation gradually lose their ability to adapt to new tasks and perform significantly worse than their freshly initialized counterparts. The main contribution of this paper is to propose a new hypothesis that experience replay addresses the loss of plasticity in continual learning. Here, experience replay is a form of memory. We provide supporting evidence for this hypothesis. In particular, we demonstrate in multiple different tasks, including regression, classification, and policy evaluation, that by simply adding an experience replay and processing the data in the experience replay with Transformers, the loss of plasticity disappears. Notably, we do not alter any standard components of deep learning. For example, we do not change backpropagation. We do not modify the activation functions. And we do not use any regularization. We conjecture that experience replay and Transformers can address the loss of plasticity because of the in-context learning phenomenon.

Problem

• Addresses loss of plasticity in continual learning
• Proposes experience replay as a solution
• Uses Transformers for in-context learning

Innovation

• Experience replay prevents plasticity loss
• Transformers process replay data effectively
• No changes to standard deep learning