FeatCal: Feature Calibration for Post-Merging Models

2026-05-13

AI Summary

This work addresses the performance degradation in model merging caused by feature drift. It presents the first systematic analysis of how feature drift propagates forward through layers and arises from local misalignment, and introduces a closed-form, forward-layerwise calibration method that requires neither gradient updates, iterative optimization, nor additional modules. Leveraging only a few samples, the approach effectively suppresses feature drift. Evaluated on CLIP and GLUE benchmarks, it significantly outperforms Surgery and ProbSurgery: on CLIP-ViT-B/32, it achieves 85.5% accuracy overall and recovers 82.9% of full performance using merely eight samples per task, while offering approximately four times faster calibration.

Abstract

Model merging combines task experts into one model and avoids joint training, retraining, or deploying many expert models, but the merged model often still underperforms task experts. We study this performance gap through feature drift, the difference between features produced by the merged model and by the expert on the same input. Our theory decomposes this drift into upstream propagation and local mismatch, tracks how it propagates and combines through later layers in forward order, and links final feature drift to output drift. This view motivates FeatCal, which uses a small calibration set to calibrate the merged model weights layer by layer in forward order, reducing feature drift while staying close to merged weights and preserving the benefits of model merging. FeatCal uses an efficient closed-form solution to update model weights, with no gradient descent, iterative optimization, or extra modules. On the main CLIP and GLUE benchmarks, FeatCal beats Surgery and ProbSurgery, the closest post-merging calibration baselines: 85.5% vs. 77.0%/78.8% on CLIP-ViT-B/32 Task Arithmetic (TA) and 85.2% vs. 83.7%/82.2% on FLAN-T5-base GLUE. On CLIP-ViT-B/32, 8 examples per task reach 82.9%, and 256 examples per task take 53 seconds, about 4x faster than both baselines, showing better sample efficiency and lower calibration cost.

Problem

• model merging
• feature drift
• performance gap
• task experts
• post-merging calibration

Innovation

• feature calibration
• model merging
• feature drift
• closed-form solution
• post-merging optimization