Model Merging in Pre-training of Large Language Models

2025-05-17

AI Summary

This work presents the first systematic exploration of model merging techniques embedded within large language model (LLM) pretraining. Addressing the lack of theoretical grounding and empirical validation for checkpoint fusion during pretraining, we propose a pretraining-oriented weight interpolation and gradient-aware fusion algorithm, supporting both Dense and Mixture-of-Experts (MoE) architectures across parameter scales from millions to tens of billions, and enabling cross-stage checkpoint fusion. We further establish a predictive model linking fusion outcomes to learning rate annealing behavior. Key contributions include: (1) demonstrating seamless integration of model fusion into the full pretraining pipeline; (2) deriving principled guidelines for fusion strategy and hyperparameter design; (3) achieving an average +1.8% improvement in downstream task performance across multi-scale models, with up to 37% reduction in training cost; and (4) open-sourcing a reproducible, practical guide for pretraining-aware model fusion.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Model merging has emerged as a promising technique for enhancing large language models, though its application in large-scale pre-training remains relatively unexplored. In this paper, we present a comprehensive investigation of model merging techniques during the pre-training process. Through extensive experiments with both dense and Mixture-of-Experts (MoE) architectures ranging from millions to over 100 billion parameters, we demonstrate that merging checkpoints trained with constant learning rates not only achieves significant performance improvements but also enables accurate prediction of annealing behavior. These improvements lead to both more efficient model development and significantly lower training costs. Our detailed ablation studies on merging strategies and hyperparameters provide new insights into the underlying mechanisms while uncovering novel applications. Through comprehensive experimental analysis, we offer the open-source community practical pre-training guidelines for effective model merging.

Problem

• Explores model merging in large-scale pre-training of LLMs
• Investigates performance gains from merging constant-LR checkpoints
• Provides guidelines for efficient model merging strategies

Innovation

• Merging checkpoints with constant learning rates
• Applying merging to dense and MoE architectures
• Providing practical pre-training guidelines