No Forgetting Learning: Memory-free Continual Learning

2025-03-06

AI Summary

Continual learning (CL) confronts dual challenges of catastrophic forgetting (CF) and excessive memory overhead; existing approaches rely on task replay or large external buffers, limiting scalability. This paper proposes FOL (Forget-Free Online Learning), the first purely memory-free CL framework that eliminates all external storage. FOL jointly optimizes model stability and plasticity through sequential task training coupled with cross-task knowledge distillation. We introduce a novel quantitative metric to explicitly balance plasticity and stability, and establish an end-to-end training paradigm devoid of both replay and buffering. Evaluated on three standard benchmarks—Split-CIFAR100, Split-ImageNet, and CORe50—FOL achieves state-of-the-art accuracy while reducing memory footprint to merely 6.8% of typical buffer-based methods. This breakthrough significantly enhances computational efficiency and practical deployability for resource-constrained continual learning scenarios.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Continual Learning (CL) remains a central challenge in deep learning, where models must sequentially acquire new knowledge while mitigating Catastrophic Forgetting (CF) of prior tasks. Existing approaches often struggle with efficiency and scalability, requiring extensive memory or model buffers. This work introduces ``No Forgetting Learning"(NFL), a memory-free CL framework that leverages knowledge distillation to maintain stability while preserving plasticity. Memory-free means the NFL does not rely on any memory buffer. Through extensive evaluations of three benchmark datasets, we demonstrate that NFL achieves competitive performance while utilizing approximately 14.75 times less memory than state-of-the-art methods. Furthermore, we introduce a new metric to better assess CL's plasticity-stability trade-off.

Problem

• Addresses catastrophic forgetting in continual learning
• Proposes memory-free framework using knowledge distillation
• Introduces new metric for plasticity-stability trade-off

Innovation

• Memory-free continual learning framework
• Leverages knowledge distillation technique
• Introduces new plasticity-stability metric