Toward a Holistic Approach to Continual Model Merging

2025-09-27

AI Summary

This paper addresses two key challenges in continual model merging—poor scalability and catastrophic forgetting—by proposing the first end-to-end controllable, replay-free framework. Methodologically: (1) prior to merging, tangent-space fine-tuning enhances task-wise weight decoupling; (2) during merging, optimizer states are leveraged to implicitly preserve function-level behavioral information, replacing naive parameter averaging; (3) post-merging, a representation correction mechanism enforces feature-space alignment. The entire pipeline operates with constant memory overhead, requiring neither historical data nor replay buffers. Evaluated on class-incremental and domain-incremental benchmarks, our approach significantly outperforms state-of-the-art methods—reducing forgetting by up to 42%—and scales robustly to hundred-task continual fusion. The framework combines theoretical rigor with practical deployability, offering a principled solution for scalable, memory-efficient continual model integration.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Search and Optimization: Mixed Discrete/Continuous Search
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

We present a holistic framework for continual model merging that intervenes at three critical stages: pre-merging, during merging, and post-merging-to address two fundamental challenges in continual learning. In particular, conventional approaches either maintain a growing list of per-domain task vectors, leading to scalability issues or rely solely on weight-space merging when old data is inaccessible, thereby losing crucial functional information. Our method overcomes these limitations by first fine-tuning the main model within its tangent space on domain-specific data; this linearization amplifies per-task weight disentanglement, effectively mitigating across-task interference. During merging, we leverage functional information from available optimizer states beyond mere parameter averages to avoid the need to revisit old data. Finally, a post-merging correction aligns the representation discrepancy between pre- and post-merged models, reducing bias and enhancing overall performance-all while operating under constant memory constraints without accessing historical data. Extensive experiments on standard class-incremental and domain-incremental benchmarks demonstrate that our approach not only achieves competitive performance but also provides a scalable and efficient solution to the catastrophic forgetting problem.

Problem

• Addresses scalability issues in continual learning model merging
• Mitigates catastrophic forgetting without accessing historical data
• Enhances model performance under constant memory constraints

Innovation

• Fine-tunes model in tangent space for task disentanglement
• Uses optimizer states beyond parameters during merging
• Applies post-merging correction to align representation discrepancies