Continual Learning Beyond Experience Rehearsal and Full Model Surrogates

2025-05-28

AI Summary

In continual learning (CL), deep neural networks suffer from catastrophic forgetting (CF), and existing approaches—relying on experience replay or full-model proxies—incur high memory and computational overhead. This paper proposes SPARC, a parameter-efficient CL framework that eliminates both replay and full-model proxies. Its core innovations include: (i) decoupling task-specific working memory from task-agnostic semantic memory to enable cross-task knowledge integration; (ii) classifier-layer weight renormalization to mitigate task bias; and (iii) a parameter-efficient knowledge consolidation architecture. SPARC achieves performance comparable to full-model proxy baselines using only 6% of their parameters. On Seq-TinyImageNet, it significantly outperforms standard CL baselines; on mainstream CL benchmarks (e.g., Split-CIFAR100, Seq-ImageNet), it matches or exceeds replay-based methods in accuracy while requiring substantially lower memory and computation. Thus, SPARC delivers high accuracy, minimal memory footprint, and low computational cost—advancing the frontier of efficient continual learning.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Continual learning (CL) has remained a significant challenge for deep neural networks as learning new tasks erases previously acquired knowledge, either partially or completely. Existing solutions often rely on experience rehearsal or full model surrogates to mitigate CF. While effective, these approaches introduce substantial memory and computational overhead, limiting their scalability and applicability in real-world scenarios. To address this, we propose SPARC, a scalable CL approach that eliminates the need for experience rehearsal and full-model surrogates. By effectively combining task-specific working memories and task-agnostic semantic memory for cross-task knowledge consolidation, SPARC results in a remarkable parameter efficiency, using only 6% of the parameters required by full-model surrogates. Despite its lightweight design, SPARC achieves superior performance on Seq-TinyImageNet and matches rehearsal-based methods on various CL benchmarks. Additionally, weight re-normalization in the classification layer mitigates task-specific biases, establishing SPARC as a practical and scalable solution for CL under stringent efficiency constraints.

Problem

• Addresses catastrophic forgetting in continual learning without rehearsal
• Reduces memory and computational overhead in scalable CL
• Mitigates task-specific biases with lightweight parameter efficiency

Innovation

• SPARC combines task-specific and semantic memories
• SPARC uses only 6% of surrogate parameters
• Weight re-normalization mitigates task-specific biases