Are we Merging the Right Models? Impact of Expert Training Duration on Model Merging for LLMs

2026-07-13

AI Summary

This work challenges the conventional practice of merging models at the point of optimal validation loss by systematically investigating how the training duration of expert models affects the performance of merged large language models. Through multi-stage training checkpoints across five domains and three model scales, evaluated with five merging methods, the study reveals a strong dependence between merging efficacy and training length. It finds that simple averaging suffers significant degradation during overfitting, whereas sparse merging methods achieve peak performance well beyond the validation-optimal step. Drawing a theoretical analogy to random forests via bias-variance decomposition, the paper proposes jointly optimizing training duration and merging strategy, establishing a new paradigm for efficient model merging.

Abstract

Multi-task model merging combines separately trained expert models into a single model that handles all tasks without co-training. Standard practice merges experts at their optimal validation loss. We challenge this convention by systematically studying how training duration of domain experts affects the quality of the merged model. We fine-tune experts on five domains (Math, Code, Instruction Following, Multilingual, and Safety) across three model sizes (Qwen 3.5 0.8B, 2B, and 4B), saving checkpoints from 25\% to 500\% of the optimal training steps and evaluating five merging methods at each duration. Our findings reveal a striking method-dependent pattern: simple averaging degrades sharply with overfitting, while sparsification-based methods achieve their best performance well past the validation optimum. We formalize this through bias-variance decomposition analysis, drawing a parallel to random forests where averaging benefits from high-variance individual learners. These results suggest that training duration and merging method should be chosen jointly rather than independently.

Problem

• model merging
• training duration
• large language models
• expert models
• validation loss

Innovation

• model merging
• training duration
• bias-variance decomposition
• expert models
• large language models