KeepLoRA: Continual Learning with Residual Gradient Adaptation

2026-01-27

AI Summary

This work addresses the challenge of balancing the retention of pre-trained knowledge and previously learned tasks while effectively acquiring new capabilities in continual learning of vision-language models. The authors propose a subspace disentanglement approach that leverages principal component analysis to decompose the parameter space into a shared principal subspace and task-specific residual subspaces. Low-rank adaptation (LoRA) is applied within the residual subspaces, and a residual gradient adaptation mechanism is introduced to project gradients of new tasks onto directions orthogonal to both the principal subspace and features of historical tasks. This design enables effective disentanglement among pre-trained, past-task, and new-task knowledge. The method achieves state-of-the-art performance across multiple continual learning benchmarks, significantly improving the trade-off between knowledge retention and adaptation to new tasks.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Computer Vision: Large Vision Models
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Continual learning for pre-trained vision-language models requires balancing three competing objectives: retaining pre-trained knowledge, preserving knowledge from a sequence of learned tasks, and maintaining the plasticity to acquire new knowledge. This paper presents a simple but effective approach called KeepLoRA to effectively balance these objectives. We first analyze the knowledge retention mechanism within the model parameter space and find that general knowledge is mainly encoded in the principal subspace, while task-specific knowledge is encoded in the residual subspace. Motivated by this finding, KeepLoRA learns new tasks by restricting LoRA parameter updates in the residual subspace to prevent interfering with previously learned capabilities. Specifically, we infuse knowledge for a new task by projecting its gradient onto a subspace orthogonal to both the principal subspace of pre-trained model and the dominant directions of previous task features. Our theoretical and empirical analyses confirm that KeepLoRA balances the three objectives and achieves state-of-the-art performance. The implementation code is available at https://github.com/MaolinLuo/KeepLoRA.

Problem

• continual learning
• vision-language models
• knowledge retention
• plasticity
• task interference

Innovation

• Continual Learning
• LoRA
• Residual Subspace
• Knowledge Retention
• Gradient Projection