JumpLoRA: Sparse Adapters for Continual Learning in Large Language Models

2026-04-17

AI Summary

This work addresses the catastrophic forgetting problem in large language models during continual learning, which arises from sequential task acquisition. To mitigate interference across tasks, the authors propose JumpLoRA, a novel framework that introduces the JumpReLU gating mechanism into the LoRA module for the first time. This design enables adaptive, sparse activation to achieve dynamic parameter isolation, effectively reducing cross-task interference. JumpLoRA maintains high modularity and compatibility while incurring minimal computational overhead, significantly outperforming state-of-the-art methods such as ELLA. The approach establishes a new paradigm for efficient and scalable continual learning in large language models.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Adapter-based methods have become a cost-effective approach to continual learning (CL) for Large Language Models (LLMs), by sequentially learning a low-rank update matrix for each task. To mitigate catastrophic forgetting, state-of-the-art approaches impose constraints on new adapters with respect to the previous ones, by targeting either subspace or coordinate-wise interference. In this paper, we propose JumpLoRA, a novel framework to adaptively induce sparsity in the Low-Rank Adaptation (LoRA) blocks through the use of JumpReLU gating. The method achieves dynamic parameter isolation, which helps prevent task interference. We demonstrate that our method is highly modular and compatible with LoRA-based CL approaches. Specifically, it significantly boosts the performance of IncLoRA and outperforms the leading state-of-the-art CL method, ELLA.

Problem

• Continual Learning
• Catastrophic Forgetting
• Large Language Models
• Adapter-based Methods

Innovation

• JumpLoRA
• Sparse Adapters
• Continual Learning
• Low-Rank Adaptation
• JumpReLU