K-Merge: Online Continual Merging of Adapters for On-device Large Language Models

2025-10-15

AI Summary

To address catastrophic forgetting and storage constraints in deploying large language models (LLMs) on memory-limited mobile devices—where multiple task-specific low-rank adapters (LoRAs) must be continuously integrated online—this paper proposes the first data-agnostic, computationally efficient online continual merging framework. Our method dynamically selects and weights newly arrived LoRAs based on importance scores, requiring only a bounded cache of adapters and eliminating the need for original training data or backpropagation. The core contribution is a lightweight online merging strategy that enables incremental adapter integration under strict memory budgets. Evaluated on multi-task continual learning benchmarks, our approach significantly outperforms existing merging methods while preserving stable performance across all historical tasks, demonstrating its effectiveness and practicality for edge deployment.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

On-device deployment of Large Language Models (LLMs) frequently leverages Low-Rank Adapters (LoRAs) to support diverse downstream tasks under tight resource constraints. To address the limited storage capacity of mobile devices, recent works have explored model merging techniques to fuse multiple LoRAs into a single one. In practice, however, LoRAs are often delivered incrementally, as users request support for new tasks (e.g., novel problem types or languages). This scenario introduces a new challenge: on-device online continual merging, where the objective is to incorporate new LoRAs while preserving the performance on previously supported tasks. In this paper, we propose a data-free and computationally efficient strategy for selecting and merging LoRAs when a new one becomes available, assuming the device can store only a limited number of adapters. Extensive experiments across real-world tasks demonstrate the superiority of our approach compared to alternative strategies while adhering to the storage budget and compute limitations of on-device settings.

Problem

• Online continual merging of LoRAs for on-device LLMs
• Preserving task performance while incorporating new adapters
• Selecting adapters under strict storage and compute constraints

Innovation

• Online continual merging of LoRA adapters
• Data-free and computationally efficient strategy
• Preserves performance under storage constraints