BID-LoRA: A Parameter-Efficient Framework for Continual Learning and Unlearning

2026-04-14

AI Summary

This work addresses the challenge in continual learning and machine unlearning of simultaneously achieving precise deletion of outdated knowledge, efficient acquisition of new knowledge, and robust preservation of retained knowledge—tasks that often lead to cross-iteration knowledge leakage. To this end, the authors propose a unified Continual Learning and Unlearning (CLU) paradigm that, for the first time, decouples knowledge retention, acquisition, and forgetting into three distinct pathways. They introduce a tri-path low-rank adaptation mechanism built upon a Bi-Directional Low-Rank Adaptation (BID-LoRA) architecture applied to attention layers, coupled with an escape-based forgetting strategy to enhance unlearning efficacy. By maximizing embedding space distances, the method achieves effective knowledge removal while updating only 5% of model parameters, significantly mitigating knowledge leakage. Extensive experiments on CIFAR-100 and CASIA-Face100 demonstrate substantial improvements over baseline methods, validating its practical utility in real-world applications such as identity management.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Recent advances in deep learning underscore the need for systems that can not only acquire new knowledge through Continual Learning (CL) but also remove outdated, sensitive, or private information through Machine Unlearning (MU). However, while CL methods are well-developed, MU techniques remain in early stages, creating a critical gap for unified frameworks that depend on both capabilities. We find that naively combining existing CL and MU approaches results in knowledge leakage a gradual degradation of foundational knowledge across repeated adaptation cycles. To address this, we formalize Continual Learning Unlearning (CLU) as a unified paradigm with three key goals: (i) precise deletion of unwanted knowledge, (ii) efficient integration of new knowledge while preserving prior information, and (iii) minimizing knowledge leakage across cycles. We propose Bi-Directional Low-Rank Adaptation (BID-LoRA), a novel framework featuring three dedicated adapter pathways-retain, new, and unlearn applied to attention layers, combined with escape unlearning that pushes forget-class embeddings to positions maximally distant from retained knowledge, updating only 5% of parameters. Experiments on CIFAR-100 show that BID-LoRA outperforms CLU baselines across multiple adaptation cycles. We further evaluate on CASIA-Face100, a curated face recognition subset, demonstrating practical applicability to real-world identity management systems where new users must be enrolled and withdrawn users removed.

Problem

• Continual Learning
• Machine Unlearning
• Knowledge Leakage
• Parameter-Efficient Adaptation
• Unified Learning Framework

Innovation

• Continual Learning Unlearning
• Parameter-Efficient Adaptation
• Low-Rank Adaptation
• Machine Unlearning
• Knowledge Leakage