PivotMerge: Bridging Heterogeneous Multimodal Pre-training via Post-Alignment Model Merging

2026-04-18

AI Summary

This work addresses the challenge of effectively integrating heterogeneous data sources in multimodal pretraining, where existing methods often suffer from suboptimal cross-modal alignment due to parameter interference and uneven layer-wise contributions. To overcome these limitations, the paper introduces model fusion into the multimodal pretraining phase for the first time, proposing a post-alignment fusion framework. This framework decouples generic alignment patterns from domain-specific variations through shared-space decomposition and a filtering mechanism, and further incorporates an alignment-guided hierarchical fusion strategy that dynamically assigns fusion weights across layers. The approach substantially mitigates cross-domain parameter conflicts, outperforms state-of-the-art methods on multiple multimodal benchmarks, and demonstrates exceptional performance and generalization capability in a post-alignment setting constructed from the CC12M dataset.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Multimodal Large Language Models (MLLMs) rely on multimodal pre-training over diverse data sources, where different datasets often induce complementary cross-modal alignment capabilities. Model merging provides a cost-effective mechanism for integrating multiple expert MLLMs with complementary strengths into a unified model. However, existing model merging research mainly focuses on post-finetuning scenarios, leaving the pre-training stage largely unexplored. We argue that the core of MLLM pre-training lies in establishing effective cross-modal alignment, which bridges visual and textual representations into a unified semantic space. Motivated by this insight, we introduce the post-alignment merging task, which aims to integrate cross-modal alignment capabilities learned from heterogeneous multimodal pre-training. This setting introduces two key challenges: cross-domain parameter interference, where parameter updates learned from different data distributions conflict during merging, and layer-wise alignment contribution disparity, where different layers and projectors contribute unevenly to cross-modal alignment. To address them, we propose \textbf{PivotMerge}, a post-alignment merging framework for cross-modal projectors. PivotMerge incorporates two key components: Shared-space Decomposition and Filtering, which disentangles shared alignment patterns from domain-specific variations and suppresses conflicting directions, and Alignment-guided Layer-wise Merging, which assigns layer-specific merging weights based on differing alignment contributions. We construct systematic CC12M-based post-alignment merging scenarios for evaluation. Extensive experiments on multiple multimodal benchmarks show that PivotMerge consistently outperforms existing baselines, demonstrating its effectiveness and generalization ability.

Problem

• multimodal pre-training
• cross-modal alignment
• model merging
• parameter interference
• layer-wise contribution

Innovation

• model merging
• multimodal pre-training
• cross-modal alignment
• post-alignment
• PivotMerge