When Shared Knowledge Hurts: Spectral Over-Accumulation in Model Merging

2026-02-05

AI Summary

This work identifies and quantifies a previously overlooked issue in model merging—termed “spectral over-accumulation”—where knowledge sharing across tasks leads to redundant accumulation along dominant spectral directions, causing singular value inflation and biasing the merged model toward a shared subspace. To address this, the authors propose a training- and data-free post-processing method that leverages singular value decomposition to analyze the weight spectrum and calibrates singular values via a subspace overlap-aware rescaling strategy, thereby restoring spectral balance. Evaluated on vision and language benchmarks, the approach significantly enhances existing merging techniques, improving the accuracy of Task Arithmetic by 13.0% and achieving state-of-the-art performance.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Model merging combines multiple fine-tuned models into a single model by adding their weight updates, providing a lightweight alternative to retraining. Existing methods primarily target resolving conflicts between task updates, leaving the failure mode of over-counting shared knowledge unaddressed. We show that when tasks share aligned spectral directions (i.e., overlapping singular vectors), a simple linear combination repeatedly accumulates these directions, inflating the singular values and biasing the merged model toward shared subspaces. To mitigate this issue, we propose Singular Value Calibration (SVC), a training-free and data-free post-processing method that quantifies subspace overlap and rescales inflated singular values to restore a balanced spectrum. Across vision and language benchmarks, SVC consistently improves strong merging baselines and achieves state-of-the-art performance. Furthermore, by modifying only the singular values, SVC improves the performance of Task Arithmetic by 13.0%. Code is available at: https://github.com/lyymuwu/SVC.

Problem

• model merging
• spectral over-accumulation
• shared knowledge
• singular values
• subspace overlap

Innovation

• model merging
• spectral over-accumulation
• singular value calibration
• subspace overlap
• task arithmetic