KeepLoRA++: Continual Learning with Layer-Scaled Residual Gradient Adaptation

2026-06-15

AI Summary

This work addresses the challenge of balancing pre-trained knowledge retention, memory of past tasks, and adaptation to new tasks in continual learning with vision–language models. The authors propose a dual-dimensional knowledge preservation mechanism grounded in an analysis of intra- and inter-layer knowledge distribution in Transformers, revealing that shallow layers encode general-purpose knowledge in principal subspaces, while deeper layers accommodate task-specific adaptations in residual subspaces. Accordingly, they introduce a layer-scaled residual gradient adaptation strategy that confines parameter updates for new tasks to residual subspaces orthogonal to both the pre-trained principal subspace and historical task features, with update magnitudes scaled smaller in shallow layers and larger in deeper ones. Integrated with LoRA fine-tuning, subspace projection, and gradient orthogonalization, the method significantly outperforms existing baselines across image classification, visual question answering, and video understanding tasks, effectively harmonizing knowledge retention with new-task learning.

Abstract

Continual learning for pre-trained vision-language models requires balancing three competing objectives: retaining pre-trained knowledge, preserving knowledge from a sequence of learned tasks, and maintaining the plasticity to acquire new knowledge. This paper presents KeepLoRA++, balancing these objectives through a unified dual-dimensional knowledge retention mechanism. We analyze knowledge distribution of Transformer architecture from both inter-layer and intra-layer perspectives. The inter-layer perspective examines how retention is distributed across layers, while the intra-layer perspective focuses on the parameter space within each layer. Our analysis reveals a structural property: general transferable knowledge is mainly encoded in the shallow layers and the principal subspace of the parameters, while task-specific adaptations are localized in the deep layers and the residual subspace. Motivated by this insight, KeepLoRA++ introduces a layer-scaled residual gradient adaptation method. New tasks are learned by restricting LoRA parameter updates to the residual subspace, combined with a shallow-to-deep layer scaling, to prevent interference with previously acquired capabilities. Specifically, the gradient of a new task is projected onto a subspace orthogonal to both the principal subspace of the pre-trained model and the dominant directions of previous task features, while simultaneously assigning smaller update magnitudes to shallow layers and larger ones to deeper layers. Our theoretical analysis and empirical evaluations confirm that KeepLoRA++ successfully balances these three competing objectives, consistently outperforming representative baselines across image classification, visual question answering, and video understanding tasks.

Problem

• continual learning
• vision-language models
• knowledge retention
• plasticity
• catastrophic forgetting

Innovation

• Continual Learning
• LoRA
• Subspace Adaptation
• Vision-Language Models
• Gradient Projection