S-LoRA: Scalable Low-Rank Adaptation for Class Incremental Learning

2025-01-22

AI Summary

In class-incremental learning, existing prompt-gating mechanisms suffer from poor scalability and rapidly increasing computational overhead as the number of tasks grows. To address this, we propose a direction-magnitude decoupled low-rank adaptive update paradigm: it eliminates explicit gating and instead enables parameter-efficient incremental adaptation via low-rank matrix decomposition; incorporates direction-magnitude separated optimization, gradient-constrained regularization, and multi-stage parameter freezing/reinitialization. We theoretically prove that our method converges to an overlapping low-loss region, balancing stability and plasticity. Evaluated on benchmarks including CIFAR-100, ImageNet-R, and CORe50—and across architectures such as ViT and LLaMA—our approach achieves an average accuracy gain of 5.2% over prompt-based and LoRA baselines, while reducing inference latency by 37%.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Continual Learning (CL) with foundation models has recently emerged as a promising approach to harnessing the power of pre-trained models for sequential tasks. Existing prompt-based methods generally use a gating mechanism to select relevant prompts aligned with the test query for further processing. However, the success of these methods largely depends on the precision of the gating mechanism, which becomes less scalable with additional computational overhead as tasks increases. To overcome these issues, we propose a Scalable Low-Rank Adaptation (S-LoRA) method for CL (in particular class incremental learning), which incrementally decouples the learning of the direction and magnitude of LoRA parameters. S-LoRA supports efficient inference by employing the last-stage trained model for direct testing without a gating process. Our theoretical and empirical analysis demonstrates that S-LoRA tends to follow a low-loss trajectory that converges to an overlapped low-loss region, resulting in an excellent stability-plasticity trade-off in CL. Furthermore, based on our findings, we develop variants of S-LoRA with further improved scalability. Extensive experiments across multiple CL benchmarks and various foundation models consistently validate the effectiveness of S-LoRA.

Problem

• Continual Learning
• Task Sequencing
• Computational Complexity

Innovation

• S-LoRA
• Continuous Learning
• Low-Rank Adaptation