Revisiting Weight Regularization for Low-Rank Continual Learning

2026-02-19

AI Summary

This work addresses the dual challenges of inter-task interference and computational overhead in parameter-efficient continual learning (PECL). The authors propose integrating Elastic Weight Consolidation (EWC) regularization into the Low-Rank Adaptation (LoRA) framework by imposing importance-based constraints on shared low-rank updates. This approach effectively balances model stability and plasticity without increasing parameter count or inference cost. The key innovation lies in leveraging low-rank representations to efficiently approximate full-dimensional parameter importance, enabling lightweight weight regularization. Experimental results demonstrate that the method significantly outperforms existing low-rank approaches across multiple continual learning benchmarks, achieving a superior trade-off between stability and plasticity.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Search and Optimization: Learning to Search
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Continual Learning (CL) with large-scale pre-trained models (PTMs) has recently gained wide attention, shifting the focus from training from scratch to continually adapting PTMs. This has given rise to a promising paradigm: parameter-efficient continual learning (PECL), where task interference is typically mitigated by assigning a task-specific module during training, such as low-rank adapters. However, weight regularization techniques, such as Elastic Weight Consolidation (EWC)-a key strategy in CL-remain underexplored in this new paradigm. In this paper, we revisit weight regularization in low-rank CL as a new perspective for mitigating task interference in PECL. Unlike existing low-rank CL methods, we mitigate task interference by regularizing a shared low-rank update through EWC, thereby keeping the storage requirement and inference costs constant regardless of the number of tasks. Our proposed method EWC-LoRA leverages a low-rank representation to estimate parameter importance over the full-dimensional space. This design offers a practical, computational- and memory-efficient solution for CL with PTMs, and provides insights that may inform the broader application of regularization techniques within PECL. Extensive experiments on various benchmarks demonstrate the effectiveness of EWC-LoRA, achieving a stability-plasticity trade-off superior to existing low-rank CL approaches. These results indicate that, even under low-rank parameterizations, weight regularization remains an effective mechanism for mitigating task interference. Code is available at: https://github.com/yaoyz96/low-rank-cl.

Problem

• Continual Learning
• Weight Regularization
• Low-Rank Adaptation
• Task Interference
• Parameter-Efficient Learning

Innovation

• weight regularization
• low-rank adaptation
• continual learning
• parameter-efficient learning
• EWC-LoRA