Self-Controlled Dynamic Expansion Model for Continual Learning

2025-04-14

AI Summary

In continual learning (CL), catastrophic forgetting and negative transfer arise due to inaccessible historical data and the rigidity of static Vision Transformer (ViT) backbones in adapting to cross-domain tasks, leading to parameter redundancy. To address this, we propose the Self-Controlled Dynamic Expansion Model (SCDEM). SCDEM synergistically integrates multiple heterogeneous pre-trained ViT backbones and introduces three core components: (i) a Collaborative Optimization Mechanism (COM) for joint backbone refinement; (ii) Optimal Transport-based Feature Distribution Consistency Alignment (FDC) to mitigate inter-task distribution shift; and (iii) a Dynamic Layer-Wise Feature Attention Mechanism (DLWFAM) for adaptive feature reweighting across layers. For the first time, SCDEM unifies multi-backbone collaboration, distribution alignment, and layer-aware attention within a self-controlled dynamic architecture. Evaluated on multiple CL benchmarks, SCDEM achieves state-of-the-art performance, significantly alleviates catastrophic forgetting, improves knowledge retention, and enhances cross-task generalization.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Large Vision Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Continual Learning (CL) epitomizes an advanced training paradigm wherein prior data samples remain inaccessible during the acquisition of new tasks. Numerous investigations have delved into leveraging a pre-trained Vision Transformer (ViT) to enhance model efficacy in continual learning. Nonetheless, these approaches typically utilize a singular, static backbone, which inadequately adapts to novel tasks, particularly when engaging with diverse data domains, due to a substantial number of inactive parameters. This paper addresses this limitation by introducing an innovative Self-Controlled Dynamic Expansion Model (SCDEM), which orchestrates multiple distinct trainable pre-trained ViT backbones to furnish diverse and semantically enriched representations. Specifically, by employing the multi-backbone architecture as a shared module, the proposed SCDEM dynamically generates a new expert with minimal parameters to accommodate a new task. A novel Collaborative Optimization Mechanism (COM) is introduced to synergistically optimize multiple backbones by harnessing prediction signals from historical experts, thereby facilitating new task learning without erasing previously acquired knowledge. Additionally, a novel Feature Distribution Consistency (FDC) approach is proposed to align semantic similarity between previously and currently learned representations through an optimal transport distance-based mechanism, effectively mitigating negative knowledge transfer effects. Furthermore, to alleviate over-regularization challenges, this paper presents a novel Dynamic Layer-Wise Feature Attention Mechanism (DLWFAM) to autonomously determine the penalization intensity on each trainable representation layer. An extensive series of experiments have been conducted to evaluate the proposed methodology's efficacy, with empirical results corroborating that the approach attains state-of-the-art performance.

Problem

• Dynamic expansion model adapts to new tasks in continual learning
• Multiple ViT backbones optimize knowledge transfer without forgetting
• Feature alignment and attention mitigate negative transfer and over-regularization

Innovation

• Dynamic multi-backbone ViT for diverse representations
• Collaborative Optimization Mechanism for expert synergy
• Feature Distribution Consistency to align semantic similarity