Model Merging as Probabilistic Inference in Fine-Tuning Parameter Space

2026-07-02

AI Summary

This work addresses the challenge of effectively merging multiple single-task models into a unified multitask model without relying on additional fine-tuning data, while rigorously evaluating the statistical validity of task-wise update directions. The authors formulate model fusion as a probabilistic inference problem in parameter space and, for the first time, cast individual single-task models as energy-based expert models within a Product-of-Experts (PoE) framework. By identifying a critical mismatch between the implicit Gaussian assumptions in existing methods and the empirically observed heavy-tailed residual distributions, they propose a heavy-tailed PoE based on the Cauchy distribution. This approach consistently outperforms current fusion strategies across diverse tasks and architectures, demonstrating that heavy-tailed modeling is essential for enhancing fusion performance.

Abstract

Model merging aims to combine existing single-task solutions into a multi-task solution without additional data-driven fine-tuning.~Most existing approaches achieve this using geometric properties of local solution spaces. However, such geometric views provide limited guidance for scoring how statistically useful each task-specific update direction is across tasks during merging. To address this, we formulate model merging from a new perspective of probabilistic inference under a product-of-experts (PoE) scenario where each single-task solution defines an energy-based expert model (EBM) over the merged parameters. We show that several existing model merging methods arise as special cases of our framework under energy designs that impose implicit Gaussian assumptions on directional residuals between merged and task-specific models. Empirically, we find that these residuals are often heavy-tailed which exposes a mismatch with the imposed light-tailed Gaussian structures. We address this with a heavy-tailed PoE design based on Cauchy experts, which better captures the observed residual behavior while admitting a provably convergent inference procedure. Experiments across multiple tasks and architectures show significant improvements over state-of-the-arts baselines. Our code is available at https://github.com/MinhLong210/PoE-EBM-Merging.git.

Problem

• model merging
• probabilistic inference
• product-of-experts
• energy-based models
• heavy-tailed residuals

Innovation

• model merging
• probabilistic inference
• product-of-experts
• energy-based model
• heavy-tailed distribution