Semantically-Shifted Incremental Adapter-Tuning is A Continual ViTransformer

2024-03-29

Computer Vision and Pattern Recognition

AI Summary

To address catastrophic forgetting and memory constraints in class-incremental learning (CIL), this paper proposes a sample-free, parameter-free continual optimization framework for Vision Transformers (ViTs). Methodologically: (1) it introduces a parameter-free shared adapter that enables progressive, efficient fine-tuning of the backbone without adding parameters; (2) it designs a prototype-based semantic drift estimator that dynamically resamples and updates classifier prototypes—without access to old-class images. The key contributions are twofold: first, it achieves full-lifecycle continual adapter tuning under strict zero-image-storage and zero-model-expansion constraints; second, it establishes a replay-free mechanism for preserving semantic consistency across tasks. Evaluated on five standard CIL benchmarks, the method consistently outperforms existing pre-trained-model-based approaches, achieving new state-of-the-art performance.

Technology Category

• Computer Vision: Learning & Optimization for CV
• Machine Learning: Life-Long and Continual Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Class-incremental learning (CIL) aims to enable models to continuously learn new classes while overcoming catas-trophic forgetting. The introduction of pre-trained models has brought new tuning paradigms to CIL. In this paper, we revisit different parameter-efficient tuning (PET) meth-ods within the context of continual learning. We observe that adapter tuning demonstrates superiority over prompt-based methods, even without parameter expansion in each learning session. Motivated by this, we propose incremen-tally tuning the shared adapter without imposing parameter update constraints, enhancing the learning capacity of the backbone. Additionally, we employ feature sampling from stored prototypes to retrain a unified classifier, further im-proving its performance. We estimate the semantic shift of old prototypes without access to past samples and update stored prototypes session by session. Our proposed method eliminates model expansion and avoids retaining any im-age samples. It surpasses previous pre-trained model-based CIL methods and demonstrates remarkable continual learning capabilities. Experimental results on five CIL bench-marks validate the effectiveness of our approach, achieving state-of-the-art (SOTA) performance.

Problem

• Overcoming catastrophic forgetting in class-incremental learning scenarios
• Enhancing model learning capacity without parameter expansion constraints
• Compensating semantic shift of old prototypes without past samples

Innovation

• Incrementally tunes shared adapter without constraints
• Retrains classifier using feature sampling from prototypes
• Estimates semantic shift to update stored prototypes