CLARE: Scalable Class-Incremental Continual Learning via a Sparsity-Based Framework

2026-09-15

AI Summary

为解决持续学习中的灾难性遗忘问题，CLARE通过稀疏微调框架，分两阶段识别关键参数并优化，减少任务间干扰。

Abstract

Continual learning must balance the learning of new knowledge with the retention of previously learned knowledge to incrementally learn tasks from a data stream without catastrophic forgetting. While leveraging pretrained models has significantly advanced continual learning, existing methods exhibit a scalability bottleneck when trained sequentially on many tasks, suffering from performance degradation due to inter-task interference and loss of plasticity. Inspired by evidence that sparse fine-tuning achieves performance comparable to full fine-tuning, this paper presents a novel sparsity-driven continual learning framework. Our continual learning method, termed CLARE, operates in two stages: it first identifies a sparse, task-critical parameter mask via a sparsity-inducing objective, then performs mask-constrained fine-tuning by only optimizing parameters selected by the mask. This two-stage sparse adapter mechanism enables all tasks to be accumulated within a shared adapter space while reducing destructive interference across tasks. Extensive experiments demonstrate the scalability of CLARE. On the long task-sequence benchmark Omnibenchmark-1k, CLARE outperforms strong baselines in final accuracy by a large margin, e.g, improving EASE by 4.64% and 13.34% after learning 100 tasks, respectively.

Problem

• continual learning
• catastrophic forgetting
• scalability
• inter-task interference
• plasticity

Innovation

• sparsity-driven continual learning
• sparse parameter mask
• mask-constrained fine-tuning
• shared adapter space