Competition and Attraction Improve Model Fusion

2025-08-22

AI Summary

Existing model merging methods rely on manual parameter grouping, which restricts the combinatorial search space and degrades generalization. This paper proposes M2N2—the first evolutionary algorithm-based framework for fully automated, from-scratch model merging. Its core innovations include (i) dynamic parameter boundary adjustment to adaptively refine the search space, (ii) a resource-competition-driven diversity preservation mechanism to maintain population heterogeneity, and (iii) a heuristic attraction-based selection strategy for identifying high-potential merging pairs. On MNIST, M2N2 achieves accuracy comparable to CMA-ES while being significantly more computationally efficient. Extended to language and vision generative models, M2N2 attains state-of-the-art performance across multiple downstream tasks. Crucially, it preserves key unoptimized capabilities of the constituent models—e.g., zero-shot reasoning or domain-specific robustness—thereby enhancing both merging efficiency and cross-task generalization.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Search and Optimization: Metareasoning and Metaheuristics
• Computer Vision: Diffusion Models for Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Models for Web evolution

Abstract

Model merging is a powerful technique for integrating the specialized knowledge of multiple machine learning models into a single model. However, existing methods require manually partitioning model parameters into fixed groups for merging, which restricts the exploration of potential combinations and limits performance. To overcome these limitations, we propose Model Merging of Natural Niches (M2N2), an evolutionary algorithm with three key features: (1) dynamic adjustment of merging boundaries to progressively explore a broader range of parameter combinations; (2) a diversity preservation mechanism inspired by the competition for resources in nature, to maintain a population of diverse, high-performing models that are particularly well-suited for merging; and (3) a heuristicbased attraction metric to identify the most promising pairs of models for fusion. Our experimental results demonstrate, for the first time, that model merging can be used to evolve models entirely from scratch. Specifically, we apply M2N2 to evolve MNIST classifiers from scratch and achieve performance comparable to CMA-ES, while being computationally more efficient. Furthermore, M2N2 scales to merge specialized language and image generation models, achieving state-of-the-art performance. Notably, it preserves crucial model capabilities beyond those explicitly optimized by the fitness function, highlighting its robustness and versatility. Our code is available at https://github.com/SakanaAI/natural_niches

Problem

• Dynamic adjustment of merging boundaries for broader parameter combinations
• Diversity preservation mechanism inspired by natural resource competition
• Heuristic-based attraction metric to identify promising model fusion pairs

Innovation

• Evolutionary algorithm with dynamic merging boundaries
• Diversity preservation via natural competition mechanism
• Heuristic attraction metric for optimal model pairing