CABS+: Efficient and Scalable Model Merging via Conflict-Aware Sparsification and Adaptive Weight Allocation

2026-08-13

AI Summary

This work addresses the performance degradation commonly observed in multi-task model merging, which stems from parameter conflicts and knowledge interference, compounded by the high computational cost of existing methods and their susceptibility to domination by high-performing tasks. To overcome these limitations, the authors propose an efficient, scalable, and retraining-free merging framework that mitigates interference through conflict-aware structured pruning, reduces coefficient search complexity via an adaptive weight allocation (AWA) mechanism, and balances task-wise benefits using an asymmetric fitness function. Additionally, they introduce a Relative Synergy Score (RSS) to quantify model mergeability. Evaluated across 27 datasets and five model families, the proposed method achieves overall performance gains of 16.97% and 12.93% over AdaMerging and WUDIMerging, respectively, while consuming less than 25% of the memory and accelerating the merging process by nearly fourfold.

Abstract

Model merging has recently attracted significant attention as a promising paradigm for constructing unified multi-task models without requiring additional retraining. However, parameter conflicts and knowledge interference across tasks often degrade merged-model performance. Prior work introduced Conflict-Aware and Balanced Sparsification (CABS), which reduces parameter interference through structured pruning and sequential masking. However, CABS relies on grid search to determine scaling coefficients, resulting in exponential time complexity, while its optimization objective can be dominated by high-performance tasks, leading to suboptimal overall performance. To address these limitations, we extend CABS and propose CABS+. Specifically, Adaptive Weight Allocation (AWA) optimizes merging coefficients via a gradient-free search scheme to reduce time complexity, while an asymmetric fitness function promotes more comprehensive performance gains across tasks. Moreover, we conduct a systematic empirical study of key factors influencing model merging performance and propose Relative Synergy Score (RSS) to quantify model mergeability and guide model selection. We compare CABS+ with state-of-the-art model merging methods, including CABS, AdaMerging, and WUDIMerging, across 27 datasets and 5 models covering large language, small-scale language, and vision models. Extensive experiments verify the effectiveness and efficiency of CABS+. Compared with AdaMerging and WUDIMerging, CABS+ improves overall performance by 16.97% and 12.93%, respectively, exhibits stronger stability and robustness across varying task numbers and model architectures, uses less than 25% of the GPU memory required by AdaMerging, and achieves nearly a 4x speedup in merging time over WUDIMerging.

Problem

• model merging
• parameter conflicts
• knowledge interference
• scaling coefficients
• task performance imbalance

Innovation

• Adaptive Weight Allocation
• Conflict-Aware Sparsification
• Model Merging
• Relative Synergy Score
• Asymmetric Fitness Function