Restoring Forgotten Knowledge in Non-Exemplar Class Incremental Learning through Test-Time Semantic Evolution

2025-03-21

AI Summary

Catastrophic forgetting in non-exemplar class-incremental learning (NECIL) arises due to the unavailability of old-class samples during incremental phases. Method: This work pioneers a test-time knowledge recovery paradigm, introducing a semantic evolution mechanism that requires neither old samples nor model retraining. It establishes a test-time semantic drift compensation framework: self-supervised modeling of semantic shift, analytical optimization (replacing online gradient updates), and dynamic feature calibration—all performed without rehearsal or access to historical data. Contribution/Results: The approach achieves state-of-the-art performance on CIFAR-100, TinyImageNet, and ImageNet100, significantly improving old-class accuracy under both cold-start and warm-start settings. By decoupling knowledge retention from data replay and parameter fine-tuning, it enhances both stability and plasticity, establishing a novel paradigm for NECIL.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Search and Optimization: Learning to Search
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Continual learning aims to accumulate knowledge over a data stream while mitigating catastrophic forgetting. In Non-exemplar Class Incremental Learning (NECIL), forgetting arises during incremental optimization because old classes are inaccessible, hindering the retention of prior knowledge. To solve this, previous methods struggle in achieving the stability-plasticity balance in the training stages. However, we note that the testing stage is rarely considered among them, but is promising to be a solution to forgetting. Therefore, we propose RoSE, which is a simple yet effective method that extbf{R}est extbf{o}res forgotten knowledge through test-time extbf{S}emantic extbf{E}volution. Specifically designed for minimizing forgetting, RoSE is a test-time semantic drift compensation framework that enables more accurate drift estimation in a self-supervised manner. Moreover, to avoid incomplete optimization during online testing, we derive an analytical solution as an alternative to gradient descent. We evaluate RoSE on CIFAR-100, TinyImageNet, and ImageNet100 datasets, under both cold-start and warm-start settings. Our method consistently outperforms most state-of-the-art (SOTA) methods across various scenarios, validating the potential and feasibility of test-time evolution in NECIL.

Problem

• Mitigates catastrophic forgetting in continual learning
• Addresses old class inaccessibility in NECIL
• Proposes test-time semantic evolution for knowledge retention

Innovation

• Test-time semantic drift compensation framework
• Self-supervised accurate drift estimation
• Analytical solution replaces gradient descent