Sparsity-Aware Evolution for Model Merging

2026-02-09

AI Summary

This work proposes a sparsity-aware evolutionary fusion framework to address the challenge of simultaneously achieving reliability and sparsity in model fusion. The approach introduces an iterative pruning-fusion cycle as a novel mutation operator and explicitly incorporates sparsity constraints into the evolutionary scoring function. By leveraging a sparsity-driven competition mechanism, the method induces local attraction effects that enhance structural complementarity among fused models. Experimental results demonstrate that the proposed framework significantly improves both the reliability and compactness of fused models across multiple large language model benchmarks. Furthermore, it exhibits strong compatibility with existing techniques, enabling orthogonal integration and offering broad applicability and practical utility in diverse model fusion scenarios.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Evolutionary Computation
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Models for Web evolution

Abstract

We propose a sparsity-aware evolutionary (SAE) framework for model merging that involves iterative pruning-merging cycles to act as a novel mutation operator. We incorporate the sparsity constraints into the score function, which steers the evolutionary process to favor more sparse models, in addition to other conventional performance scores. Interestingly, the by-product of \textit{competition} for sparsity introduces an extra local \textit{attraction} and interplay into the evolutionary process: if one competitor has more zero elements, the other competitor's non-zero elements will occupy those positions, even though the less sparse competitor loses to the more sparse competitor in other positions. The proposed pipeline is evaluated on a variety of large-scale LLM benchmarks. Experiments demonstrate that our approach can improve model merging reliability across multiple benchmarks, and is easy to incorporate due to its simplicity and being orthogonal to most existing approaches.

Problem

• model merging
• sparsity
• evolutionary algorithm
• large language models
• pruning

Innovation

• sparsity-aware
• evolutionary algorithm
• model merging
• iterative pruning
• sparse competition