An Empirical Study and Theoretical Explanation on Task-Level Model-Merging Collapse

2026-03-10

AI Summary

This work systematically investigates the performance collapse phenomenon observed in task-level merging of large language models. Through extensive experiments and theoretical analysis, it reveals—contrary to the prevailing assumption of parameter space conflict—that representation incompatibility is the primary cause of merging failure. Leveraging rate-distortion theory, the study establishes a dimensionality-dependent theoretical bound on mergeability and demonstrates, via representation similarity metrics and statistical validation, a strong correlation between representational compatibility and merged model performance. These findings fundamentally challenge the dominant paradigm in the field of model merging, offering a new theoretical foundation for understanding and improving the composability of large language models.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience

Abstract

Model merging unifies independently fine-tuned LLMs from the same base, enabling reuse and integration of parallel development efforts without retraining. However, in practice we observe that merging does not always succeed: certain combinations of task-specialist models suffer from catastrophic performance degradation after merging. We refer to this failure mode as merging collapse. Intuitively, collapse arises when the learned representations or parameter adjustments for different tasks are fundamentally incompatible, so that merging forces destructive interference rather than synergy. In this paper, we identify and characterize the phenomenon of task-level merging collapse, where certain task combinations consistently trigger huge performance degradation across all merging methods. Through extensive experiments and statistical analysis, we demonstrate that representational incompatibility between tasks is strongly correlated with merging collapse, while parameter-space conflict metrics show minimal correlation, challenging conventional wisdom in model merging literature. We provide a theoretical explanation on this phenomenon through rate-distortion theory with a dimension-dependent bound, establishing fundamental limits on task mergeability regardless of methodology.

Problem

• model merging
• merging collapse
• task-level incompatibility
• representational incompatibility
• large language models

Innovation

• model merging
• merging collapse
• representational incompatibility
• rate-distortion theory
• task compatibility