Parameter Efficient Continual Learning with Dynamic Low-Rank Adaptation

2025-05-17

AI Summary

Addressing the dual challenges of catastrophic forgetting and sensitivity to LoRA rank selection in continual learning, this paper proposes PEARL—a replay-free, lightweight framework. Its core innovation is a novel dynamic rank allocation mechanism based on parameter-space distance, enabling LoRA adapters to automatically adapt their rank to task-specific complexity without manual hyperparameter tuning or empirical rank pre-specification. By quantifying task dissimilarity via gradient or weight updates in parameter space, PEARL allocates higher ranks to more complex or divergent tasks while preserving compactness for simpler ones. Evaluated on ResNet, Separable CNN, and Vision Transformer backbones across diverse continual learning benchmarks—including class-incremental, domain-incremental, and task-incremental settings—PEARL consistently mitigates knowledge degradation and outperforms state-of-the-art baselines. It achieves superior parameter efficiency (e.g., <0.5% additional parameters per task) and strong generalization, demonstrating robustness across architectures and scenarios.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Computer Vision: Large Vision Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Catastrophic forgetting has remained a critical challenge for deep neural networks in Continual Learning (CL) as it undermines consolidated knowledge when learning new tasks. Parameter efficient fine tuning CL techniques are gaining traction for their effectiveness in addressing catastrophic forgetting with a lightweight training schedule while avoiding degradation of consolidated knowledge in pre-trained models. However, low rank adapters (LoRA) in these approaches are highly sensitive to rank selection which can lead to sub-optimal resource allocation and performance. To this end, we introduce PEARL, a rehearsal-free CL framework that entails dynamic rank allocation for LoRA components during CL training. Specifically, PEARL leverages reference task weights and adaptively determines the rank of task-specific LoRA components based on the current tasks' proximity to reference task weights in parameter space. To demonstrate the versatility of PEARL, we evaluate it across three vision architectures (ResNet, Separable Convolutional Network and Vision Transformer) and a multitude of CL scenarios, and show that PEARL outperforms all considered baselines by a large margin.

Problem

• Addresses catastrophic forgetting in continual learning
• Optimizes dynamic rank allocation for LoRA adapters
• Enhances parameter efficiency across multiple architectures

Innovation

• Dynamic rank allocation for LoRA components
• Reference task weights guide rank selection
• Rehearsal-free continual learning framework