Pruned Adaptation Modules: A Simple yet Strong Baseline for Continual Foundation Models

2026-03-22

AI Summary

This work addresses a critical gap in continual learning research, where foundation model approaches are often evaluated without fair comparison to lightweight convolutional baselines, obscuring genuine progress. To bridge this evaluation divide, we propose an efficient architecture that freezes a pretrained ResNet backbone and introduces only sparse, task-specific adaptation layers, enhanced with channel pruning to yield a parameter-efficient continual learning system. The resulting Pruned Adaptation Modules (PAM) establish a new baseline that outperforms state-of-the-art foundation-model-based methods across multiple benchmarks, while reducing trainable parameters by approximately 5× and total parameters by 6×. This approach significantly lowers update costs and effectively mitigates catastrophic forgetting, offering a more balanced and resource-efficient alternative that reconciles the performance disparity between traditional and foundation-model paradigms.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Large Vision Models
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems

Abstract

The continual learning literature has rapidly shifted from traditional class incremental learning (CIL) techniques to foundation model (FM)-based CIL methods without a clear understanding of how these newer approaches compare to strong, lightweight convolutional baselines. This abrupt transition has created a substantial methodological gap, making it difficult to assess whether recent FM-based CIL progress reflects genuine advances or merely the absence of rigorous baselines. To address this gap, we introduce Pruned Adaptation Modules (PAM), a simple yet effective method that freezes the vast majority of the pre-trained ResNet while enabling scalable continual adaptation through sparse task-specific layers. PAM yields up to a ~5x reduction in trainable parameters and a ~6x reduction in total parameters, significantly reducing the cost of continual updates. Across diverse benchmarks, PAM consistently mitigates catastrophic forgetting and outperforms state-of-the-art FM-based CIL approaches. Our findings position PAM as a strong and transparent baseline that helps bridge the gap between traditional and FM-based CIL, guiding future research for a more accurate assessment of true progress in continual adaptation. The code can be found at: https://github.com/ElifCerenGokYildirim/PAM.

Problem

• continual learning
• class incremental learning
• foundation models
• baselines
• catastrophic forgetting

Innovation

• Pruned Adaptation Modules
• Continual Learning
• Foundation Models
• Parameter Efficiency
• Catastrophic Forgetting