SimMerge: Learning to Select Merge Operators from Similarity Signals
This work proposes SimMerge, a novel approach to large language model merging that circumvents the costly trial-and-error evaluation typically required to select operators, subsets, and merging orders. SimMerge leverages task-agnostic inter-model similarity signals—capturing both functional and structural characteristics—to predict pairwise merging performance using only a small set of unlabeled probes. By doing so, it efficiently identifies optimal merging strategies without resorting to time-consuming merge-and-evaluate cycles. The method supports dynamic incorporation of new tasks, models, and operators, and seamlessly generalizes to multi-way merges and extremely large models (up to 111B parameters). Experiments demonstrate that SimMerge outperforms standard merging operators in pairwise 7B-model merges, substantially reducing evaluation overhead while maintaining strong performance.