A Survey of Continual Reinforcement Learning

2025-06-26

AI Summary

Reinforcement learning (RL) suffers from high data and computational costs, poor cross-task generalization, and catastrophic forgetting—key bottlenecks hindering its continual deployment in dynamic real-world environments. To address these challenges, this work systematically surveys continual reinforcement learning (CRL), proposing the first taxonomy grounded in a “knowledge storage–transfer” perspective, categorizing CRL methods into four principled classes. It unifies evaluation metrics, benchmark tasks, and scenario constraints. Through bibliometric analysis, methodological abstraction, and cross-benchmark comparative study, the survey covers dominant paradigms—including experience replay, regularization, parameter isolation, and meta-learning. Furthermore, it constructs the first structured, holistic CRL landscape, clarifying community consensus and open debates while identifying domain-specific challenges and evolutionary trajectories. The findings provide a rigorous theoretical framework and practical guidelines for algorithm design, standardized evaluation, and real-world CRL deployment.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Multiagent Systems: Multiagent Learning
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work

Abstract

Reinforcement Learning (RL) is an important machine learning paradigm for solving sequential decision-making problems. Recent years have witnessed remarkable progress in this field due to the rapid development of deep neural networks. However, the success of RL currently relies on extensive training data and computational resources. In addition, RL's limited ability to generalize across tasks restricts its applicability in dynamic and real-world environments. With the arisen of Continual Learning (CL), Continual Reinforcement Learning (CRL) has emerged as a promising research direction to address these limitations by enabling agents to learn continuously, adapt to new tasks, and retain previously acquired knowledge. In this survey, we provide a comprehensive examination of CRL, focusing on its core concepts, challenges, and methodologies. Firstly, we conduct a detailed review of existing works, organizing and analyzing their metrics, tasks, benchmarks, and scenario settings. Secondly, we propose a new taxonomy of CRL methods, categorizing them into four types from the perspective of knowledge storage and/or transfer. Finally, our analysis highlights the unique challenges of CRL and provides practical insights into future directions.

Problem

• Addresses RL's reliance on extensive training data
• Improves RL's generalization across diverse tasks
• Enables continuous learning and knowledge retention

Innovation

• Survey of continual reinforcement learning methods
• Taxonomy of CRL knowledge storage techniques
• Analysis of CRL challenges and future directions