Predicting Mergeability of Parameter-Efficient Fine-Tuning Updates

2026-06-17

AI Summary

This work addresses the significant performance degradation often observed after merging Low-Rank Adaptation (LoRA) modules, a problem exacerbated by existing approaches that can only assess merge compatibility post-training, leading to wasted computational resources. The paper introduces MergeProbe, the first method capable of predicting post-merge performance retention early in training. MergeProbe leverages lightweight analysis of the alignment between low-rank updates and original gradients, as well as their perturbation of shared representations. It supports diverse composition strategies—including merging, reweighting, pruning, and routing—and consistently outperforms existing interference-aware merging techniques on the MERGE-PEFT benchmark across five domains: mathematics, code, science, instruction following, and safety. Notably, MergeProbe achieves superior average and worst-case performance retention while incurring substantially lower deployment overhead than full task routing.

Abstract

Low-rank adaptation (LoRA) makes it cheap to train many domain- and task-specific language model adapters, but whether two adapters can be merged is usually discovered only after both have been fully trained and evaluated. This late feedback is costly: adapters that are strong in isolation can interfere destructively once their updates are combined. We ask whether this outcome can be anticipated. We formalize adapter mergeability as the degree to which an adapter preserves its single-task utility after merging, and show that it can be forecast from signals measured in the first few percent of training -- chiefly how the low-rank updates and their gradients align across tasks and how much they disturb shared representations. We package these signals into MergeProbe, a lightweight predictor that estimates pairwise and set-level retention and turns the estimate into a concrete decision: merge directly, reweight, prune, or route. On MERGE-PEFT, a five-domain benchmark spanning math, code, science, instruction following, and safety, MergeProbe attains the best average and worst-case retention among strong interference-aware merge baselines while adding far less deployment overhead than full task routing. This turns LoRA merging from a post-hoc engineering step into an anticipatory measurement problem.

Problem

• mergeability
• parameter-efficient fine-tuning
• LoRA
• adapter interference
• model merging

Innovation

• mergeability prediction
• parameter-efficient fine-tuning
• LoRA
• adapter merging
• MergeProbe