Regularizing Subspace Redundancy of Low-Rank Adaptation

2025-07-28

AI Summary

Low-rank adaptation methods (e.g., LoRA) achieve parameter efficiency but suffer from redundant subspace representations due to unconstrained projection matrices, degrading feature adaptation capability. Existing redundancy-mitigation strategies—such as manual rank tuning or implicit masking—exhibit poor generalizability and limited flexibility. Method: We establish, for the first time, a theoretical equivalence between low-rank matrices and their underlying subspaces, and propose a novel paradigm that explicitly models subspace redundancy and regularizes it adaptively. Our approach introduces learnable redundancy constraints to jointly optimize multiple projection subspaces, enabling plug-and-play deployment without inference overhead. Contribution/Results: Extensive experiments demonstrate significant improvements over state-of-the-art methods across diverse backbone architectures and tasks—including vision-language retrieval and image classification—while exhibiting strong generalizability and architecture-agnostic performance.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Computer Vision: Large Vision Models
• Search and Optimization: Non-convex Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Low-Rank Adaptation (LoRA) and its variants have delivered strong capability in Parameter-Efficient Transfer Learning (PETL) by minimizing trainable parameters and benefiting from reparameterization. However, their projection matrices remain unrestricted during training, causing high representation redundancy and diminishing the effectiveness of feature adaptation in the resulting subspaces. While existing methods mitigate this by manually adjusting the rank or implicitly applying channel-wise masks, they lack flexibility and generalize poorly across various datasets and architectures. Hence, we propose ReSoRA, a method that explicitly models redundancy between mapping subspaces and adaptively Regularizes Subspace redundancy of Low-Rank Adaptation. Specifically, it theoretically decomposes the low-rank submatrices into multiple equivalent subspaces and systematically applies de-redundancy constraints to the feature distributions across different projections. Extensive experiments validate that our proposed method consistently facilitates existing state-of-the-art PETL methods across various backbones and datasets in vision-language retrieval and standard visual classification benchmarks. Besides, as a training supervision, ReSoRA can be seamlessly integrated into existing approaches in a plug-and-play manner, with no additional inference costs. Code is publicly available at: https://github.com/Lucenova/ReSoRA.

Problem

• Reduces redundancy in LoRA subspaces during training
• Improves feature adaptation flexibility across datasets
• Enhances existing PETL methods without inference costs

Innovation

• Regularizes subspace redundancy in LoRA
• Decomposes low-rank submatrices into subspaces
• Plug-and-play with no inference cost