CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging

2025-02-26

AI Summary

To address parameter overlap and imbalanced inter-layer weight distributions caused by task vector conflicts in model merging, this paper proposes the Conflict-Aware and Balanced Sparsification (CABS) framework. CABS jointly designs two novel components: Conflict-Aware (CA) pruning, which models task vector discrepancies and applies sequential masking to reduce parameter overlap; and Balanced Sparsification (BS), which integrates *n:m* structured pruning with intra-layer weight distribution constraints to enforce cross-layer weight uniformity. Crucially, CABS requires no additional training while enhancing multi-task generalization. Evaluated on comprehensive multi-task benchmarks, it outperforms all state-of-the-art methods: parameter overlap is reduced by 37%, and average task accuracy improves by up to 4.2%. The gains are consistent across both small- and large-scale merged models, demonstrating robust scalability and effectiveness.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Computer Vision: Multi-modal Vision
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Model merging based on task vectors, i.e., the parameter differences between fine-tuned models and a shared base model, provides an efficient way to integrate multiple task-specific models into a multitask model without retraining. Recent works have endeavored to address the conflicts between task vectors, one of the significant challenges faced by model merging, through sparsification; however, two issues significantly limit their performance: high parameter overlap and unbalanced weight distribution. To address these issues, we propose a simple, yet effective framework called CABS (Conflict-Aware and Balanced Sparsification), consisting of Conflict-Aware Sparsification (CA) and Balanced Sparsification (BS). CA can reduce parameter overlap by applying masks during sequential pruning, ensuring that each task vector retains distinct, non-overlapping parameters. BS leverages $n$: $m$ pruning to preserve critical weights while maintaining an even distribution across layers. Our comprehensive experiments demonstrate that CABS outperforms state-of-the-art methods across diverse tasks and model sizes.

Problem

• Reduces parameter overlap in model merging
• Balances weight distribution across layers
• Enhances multitask model performance without retraining

Innovation

• Conflict-Aware Sparsification reduces parameter overlap.
• Balanced Sparsification ensures even weight distribution.
• CABS framework enhances model merging efficiency.