Continual Knowledge Consolidation LORA for Domain Incremental Learning

2025-10-17

AI Summary

To address severe catastrophic forgetting, inaccurate task-specific LoRA selection, and weak classifier generalization in Domain Incremental Learning (DIL), this paper proposes a Shared-Specific Collaborative Low-Rank Adaptation framework. Methodologically, it jointly models shared and task-specific LoRA modules, incorporating a knowledge consolidation mechanism to mitigate forgetting; replaces conventional linear or prototype classifiers with a distribution-aware stochastic classifier; and introduces multi-layer local classifiers with intermediate representation utilization, augmented by a sphere generator loss and transformation module to alleviate synthetic sample bias and knowledge selection mismatch. Evaluated on four mainstream DIL benchmarks, the method achieves an average accuracy improvement of over 5%, significantly enhancing both accuracy and stability in continual learning. Key contributions include: (1) a unified LoRA architecture balancing shared and task-specific adaptation; (2) a stochastic classifier grounded in feature distribution sampling; and (3) a holistic regularization strategy integrating geometric constraints and intermediate representation reuse.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Diffusion Models for Vision
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Domain Incremental Learning (DIL) is a continual learning sub-branch that aims to address never-ending arrivals of new domains without catastrophic forgetting problems. Despite the advent of parameter-efficient fine-tuning (PEFT) approaches, existing works create task-specific LoRAs overlooking shared knowledge across tasks. Inaccurate selection of task-specific LORAs during inference results in significant drops in accuracy, while existing works rely on linear or prototype-based classifiers, which have suboptimal generalization powers. Our paper proposes continual knowledge consolidation low rank adaptation (CONEC-LoRA) addressing the DIL problems. CONEC-LoRA is developed from consolidations between task-shared LORA to extract common knowledge and task-specific LORA to embrace domain-specific knowledge. Unlike existing approaches, CONEC-LoRA integrates the concept of a stochastic classifier whose parameters are sampled from a distribution, thus enhancing the likelihood of correct classifications. Last but not least, an auxiliary network is deployed to optimally predict the task-specific LoRAs for inferences and implements the concept of a different-depth network structure in which every layer is connected with a local classifier to take advantage of intermediate representations. This module integrates the ball-generator loss and transformation module to address the synthetic sample bias problem. Our rigorous experiments demonstrate the advantage of CONEC-LoRA over prior arts in 4 popular benchmark problems with over 5% margins.

Problem

• Preventing catastrophic forgetting in domain incremental learning scenarios
• Improving task-specific LoRA selection accuracy during model inference
• Enhancing generalization power beyond linear or prototype-based classifiers

Innovation

• Consolidates task-shared and task-specific LoRA modules
• Implements stochastic classifier from distribution sampling
• Uses auxiliary network with local classifiers per layer