AdaMMS: Model Merging for Heterogeneous Multimodal Large Language Models with Unsupervised Coefficient Optimization

2025-03-31

AI Summary

This work addresses the challenge of effectively fusing heterogeneous multimodal large language models (MLLMs), whose architectural and parameter-space asymmetries impede conventional fusion. We propose the first unsupervised, cross-architecture MLLM fusion framework, comprising three key components: (1) a learnable mapping function to align disparate model architectures; (2) weighted linear interpolation for parameter-space fusion; and (3) an unsupervised, task-agnostic hyperparameter search guided by validation loss to adaptively optimize fusion weights. Crucially, the method requires no labeled data and supports fusion across structurally divergent models—including Qwen-VL, LLaVA, and MiniCPM-V. Extensive experiments on multiple vision-language benchmarks demonstrate substantial improvements over existing fusion approaches, validating the framework’s effectiveness, generalizability, and practical utility.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recently, model merging methods have demonstrated powerful strengths in combining abilities on various tasks from multiple Large Language Models (LLMs). While previous model merging methods mainly focus on merging homogeneous models with identical architecture, they meet challenges when dealing with Multimodal Large Language Models (MLLMs) with inherent heterogeneous property, including differences in model architecture and the asymmetry in the parameter space. In this work, we propose AdaMMS, a novel model merging method tailored for heterogeneous MLLMs. Our method tackles the challenges in three steps: mapping, merging and searching. Specifically, we first design mapping function between models to apply model merging on MLLMs with different architecture. Then we apply linear interpolation on model weights to actively adapt the asymmetry in the heterogeneous MLLMs. Finally in the hyper-parameter searching step, we propose an unsupervised hyper-parameter selection method for model merging. As the first model merging method capable of merging heterogeneous MLLMs without labeled data, extensive experiments on various model combinations demonstrated that AdaMMS outperforms previous model merging methods on various vision-language benchmarks.

Problem

• Merging heterogeneous multimodal large language models
• Optimizing unsupervised coefficient for model merging
• Addressing architecture and parameter space asymmetry

Innovation

• Mapping function for different model architectures
• Linear interpolation for asymmetric parameter adaptation
• Unsupervised hyper-parameter selection method