iGSP:Implicit Gradient Subspace Projection for Efficient Continual Learning of Vision-Language Models

2026-05-18

AI Summary

This work addresses the challenges of parameter explosion and negative transfer in vision-language continual learning, which arise from task isolation and alignment based solely on superficial similarity. The authors reformulate shared alignment as a geometric problem of optimizing trajectory overlap within a low-rank subspace and propose an implicit gradient subspace projection mechanism. Their two-stage strategy first identifies a shared subspace to maximize knowledge reuse and then fine-tunes task-specific residuals in an orthogonal subspace, thereby circumventing conventional similarity assumptions. Leveraging the early-convergence property of MoE routers to construct subspace bases, the method integrates subspace-constrained regularization, basis pre-expansion, and routing-probability-driven dimension pruning. Evaluated on the MTIL benchmark, it achieves state-of-the-art performance while reducing trainable parameters by 42.7% and total model size by 86.9%.

Abstract

Vision-Language Models require efficient adaptation to continually emerging downstream tasks. While Parameter-Efficient Fine-Tuning mitigates catastrophic forgetting, assigning isolated modules per task leads to parameter explosion. Conversely, recent similarity-driven sharing mechanisms falsely equate superficial visual similarity with underlying alignment consistency. This fundamental mismatch triggers severe negative transfer between visually similar but logically distinct tasks and fails to exploit alignment reuse across visually diverse ones. We argue thatalignment sharing is fundamentally a geometric problem of overlapping optimization trajectories within shared low-rank subspaces. Grounded in this insight, we propose iGSP, a novel framework that achieves efficient adaptation via implicit gradient subspace projection. Leveraging the early convergence of MoE routers to establish the subspace basis, iGSP bifurcates the adaptation process into two phases. First, the Subspace Identification phase introduces candidate experts via basis pre-expansion, applies a novel subspace-constrained regularization to implicitly project new task gradients onto the historical subspace, and precisely prunes redundant dimensions by treating routing probabilities as gradient flow indicators, ultimately to maximize knowledge reuse. Second, the Orthogonal Subspace Fine-Tuning phase fixes this structural basis and removes the regularization to rapidly fit the task-specific residual loss. Extensive experiments on the MTIL benchmark demonstrate that iGSP achieves state-of-the-art accuracy while significantly improving training efficiency, reducing the average trainable parameters by 42.7\% compared to current SOTA methods, and decreasing the final total parameters by 86.9\% relative to counterparts. The source code is available at https://github.com/GeoX-Lab/iGSP.

Problem

• Continual Learning
• Vision-Language Models
• Parameter-Efficient Fine-Tuning
• Negative Transfer
• Alignment Reuse

Innovation

• Implicit Gradient Subspace Projection
• Continual Learning
• Vision-Language Models
• Parameter-Efficient Fine-Tuning
• Subspace Sharing