Multi-Objective Bayesian Optimization for Model Merging

2026-08-14

AI Summary

This study addresses the challenges of parameter selection, costly evaluation, and multi-objective conflicts in model merging by proposing MOBO-Merge, a framework decoupled from specific merging operators. The method formulates parameter search as a black-box multi-objective optimization problem, employing Multi-Objective Bayesian Optimization to efficiently approximate the Pareto frontier under limited budgets and systematically resolve capability trade-offs. Experimental results demonstrate that MOBO-Merge outperforms random search in 11 out of 12 comparative settings and achieves superior performance with operators such as TIES and Block-Linear. These findings validate the framework’s effectiveness and generalizability across complex model fusion tasks, offering a robust solution for balancing competing objectives without operator-specific tuning.

Abstract

Model merging combines trained models directly in weight space, offering a compute-efficient alternative to additional fine-tuning. Selecting merge parameters is nevertheless difficult because downstream evaluations are expensive, gradients are unavailable, and source capabilities can conflict. We formulate merge-parameter selection as a black-box multi-objective optimization problem and introduce MOBO-Merge, a merge-operator agnostic framework that uses multi-objective Bayesian optimization to approximate the Pareto front under a limited evaluation budget. We evaluate Qwen3-4B and Llama-3.1-8B in two-model instruction-math and three-model instruction-math-code settings using Linear, SLERP, TIES, and block-wise merge operators. On held-out benchmark partitions, MOBO-Merge obtains higher mean hypervolume than random search in 11 of 12 reported comparisons. The gain is small for one-dimensional Linear interpolation but substantially larger for several TIES, block-wise, and three-objective searches. No merge operator is uniformly best: TIES leads in three of four family-setting combinations, whereas Block-Linear 4x is strongest for the Llama three-model merge. These results show that multi-objective Bayesian optimization is valuable as a search layer for expressive merge parameterizations.

Problem

• Model Merging
• Merge Parameter Selection
• Multi-Objective Optimization
• Black-box Optimization

Innovation

• Multi-Objective Bayesian Optimization
• Model Merging
• MOBO-Merge
• Pareto Front Approximation
• Merge-Operator Agnostic