Tracing and Coordinating Cross-Layer Influence for Multimodal Model Merging

2026-09-11

AI Summary

本文针对多模态模型融合中跨层影响追踪与协调的问题，提出了TAC-Merge方法，通过构建更新效应图和联合优化权重来有效整合不同专家的能力。

Abstract

Multimodal model merging aims to consolidate task experts into a single model that retains their complementary capabilities. Most unimodal model merging methods combine expert updates within individual layers, and multimodal approaches largely follow this design. However, an expert update changes the representations passed to subsequent layers, allowing its influence to propagate across depth and affect how visual and textual information interact. When visual and language updates are combined, later updates act on inputs already modified by earlier ones, coupling their effects. This poses two challenges: (1) how to characterize the multimodal influence of individual expert updates across depth, and (2) how to jointly combine expert updates based on their multimodal influence. To address these challenges, we propose TAC-Merge for tracing and coordinating cross-layer influence in multimodal model merging. It contains two modules, i.e., multimodal influence mapping (MIM) and coupled merge control (CMC). MIM constructs graphs of update effects and uses Ricci curvature together with expert predictions to define a shared fusion objective. CMC models interactions among coefficient adjustments and jointly optimizes regional weights to synthesize one shared model. Experiments across diverse multimodal tasks demonstrate the effectiveness of TAC-Merge in consolidating complementary expert capabilities and supporting generalization to unseen tasks.

Problem

• multimodal model merging
• cross-layer influence
• expert updates

Innovation

• TAC-Merge
• Multimodal Influence Mapping (MIM)
• Coupled Merge Control (CMC)