🤖 AI Summary
In class-incremental learning (CIL), large pre-trained models (PTMs) face a fundamental trade-off between stability and plasticity under heterogeneous task distributions, suffer from sensitivity to data corruption, and are prone to catastrophic forgetting. To address these challenges, this paper proposes TTACIL—a novel framework that abandons per-task fine-tuning; instead, it performs only lightweight adapter tuning on the first task and thereafter employs test-time adaptation (TTA) exclusively to dynamically optimize LayerNorm parameters for task-aware representation updates. TTACIL is the first work to systematically integrate TTA into CIL, enabling zero-forgetting adaptation to novel classes without any additional training. Extensive experiments on CIFAR-100 and ImageNet-1K demonstrate significant improvements over state-of-the-art methods, while comprehensive robustness evaluations confirm strong resilience against diverse image corruptions.
📝 Abstract
Class-incremental learning (CIL) is a challenging task that involves sequentially learning to categorize classes from new tasks without forgetting previously learned information. The advent of large pre-trained models (PTMs) has fast-tracked the progress in CIL due to the highly transferable PTM representations, where tuning a small set of parameters leads to state-of-the-art performance when compared with the traditional CIL methods that are trained from scratch. However, repeated fine-tuning on each task destroys the rich representations of the PTMs and further leads to forgetting previous tasks. To strike a balance between the stability and plasticity of PTMs for CIL, we propose a novel perspective of eliminating training on every new task and instead train PTM only on the first task, and then refine its representation at inference time using test-time adaptation (TTA). Concretely, we propose Test-Time Adaptation for Class-Incremental Learning (TTACIL) that first fine-tunes PTMs using Adapters on the first task, then adjusts Layer Norm parameters of the PTM on each test instance for learning task-specific features, and finally resets them back to the adapted model to preserve stability. As a consequence, our TTACIL does not undergo any forgetting, while benefiting each task with the rich PTM features. Additionally, by design, our TTACIL is robust to common data corruptions. Our method outperforms several state-of-the-art CIL methods when evaluated on multiple CIL benchmarks under both clean and corrupted data. Code is available at: https://github.com/IemProg/TTACIL.
