Post-Hoc Merging is Not Enough: Many-Shot Model Merging with Loss-Gap Balancing

2026-06-15

AI Summary

This work proposes METIS, an iterative multi-round model merging framework that addresses the limitations of conventional post-training one-shot merging approaches, which often induce task interference and catastrophic forgetting—particularly degrading performance on the weakest tasks. Departing from the standard single-step paradigm, METIS dynamically balances contributions across tasks during merging by integrating loss-gap-based task weighting, consensus-oriented parameter masking, and a multi-round fusion mechanism. Experimental results demonstrate that METIS effectively mitigates information erasure and significantly enhances the overall performance of multitask large language models, with especially pronounced gains on the worst-performing tasks.

Abstract

Model merging has become a practical post-training strategy for building a single multi-task large language model (LLM) by combining multiple task-specialized models. However, most existing approaches rely on post-hoc merging, in which task-specific models are merged only once after training. This one-shot aggregation often suffers from task interference, leading to information erasure across individual tasks. In this work, we show that replacing post-hoc merging with an iterative many-shot merging protocol is effective in improving multi-task performance. Building on this insight, we propose METIS, Mitigating Erasure from Task Interference for Stable many-shot merging. METIS is a loss-aware many-shot merging method that addresses information erasure in post-hoc merging through task-wise loss-gap weighting and consensus-based masking. Notably, METIS exhibits significant performance improvement on the worst-performing task, effectively mitigating information erasure. (Project page: https://imkyungjin.github.io/METIS/)

Problem

• model merging
• task interference
• information erasure
• multi-task learning
• large language models

Innovation

• many-shot merging
• loss-gap balancing
• task interference
• model merging
• consensus-based masking