MedSAMix: A Training-Free Model Merging Approach for Medical Image Segmentation

2025-08-14

AI Summary

General-purpose medical image segmentation models (e.g., MedSAM) suffer from limited training data, high annotation costs, and domain distribution shifts, resulting in poor generalization. To address this, we propose MedSAMix—a zero-shot, training-free, and fine-tuning-free model fusion framework that automatically discovers layer-wise weight fusion strategies via zeroth-order optimization. MedSAMix synergistically integrates generic vision models with domain-specific medical segmentation models, preserving anatomical and pathological specificity while enhancing cross-task generalization. It supports both single-task and multi-objective optimization, offering flexibility and robustness. Extensive experiments across 25 diverse medical segmentation tasks demonstrate that MedSAMix achieves an average performance gain of 6.67% on domain-specialized tasks and a 4.37% improvement in multi-task holistic performance, significantly mitigating model bias. This work establishes an efficient, scalable paradigm for low-resource medical image segmentation.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Mixture of Experts (MoE)
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Universal medical image segmentation models have emerged as a promising paradigm due to their strong generalizability across diverse tasks, showing great potential for a wide range of clinical applications. This potential has been partly driven by the success of general-purpose vision models such as the Segment Anything Model (SAM), which has inspired the development of various fine-tuned variants for medical segmentation tasks. However, fine-tuned variants like MedSAM are trained on comparatively limited medical imaging data that often suffers from heterogeneity, scarce annotations, and distributional shifts. These challenges limit their ability to generalize across a wide range of medical segmentation tasks. In this regard, we propose MedSAMix, a training-free model merging method that integrates the strengths of both generalist models (e.g., SAM) and specialist models (e.g., MedSAM) for medical image segmentation. In contrast to traditional model merging approaches that rely on manual configuration and often result in suboptimal outcomes, we propose a zero-order optimization method to automatically discover optimal layer-wise merging solutions. Furthermore, for clinical applications, we develop two regimes to meet the demand of domain-specificity and generalizability in different scenarios by single-task optimization and multi-objective optimization respectively. Extensive evaluations on 25 medical segmentation tasks demonstrate that MedSAMix effectively mitigates model bias and consistently improves performance in both domain-specific accuracy and generalization, achieving improvements of 6.67% on specialized tasks and 4.37% on multi-task evaluations.

Problem

• Improves medical image segmentation generalization across diverse tasks
• Automates optimal model merging without manual configuration
• Enhances performance in domain-specific and multi-task scenarios

Innovation

• Training-free model merging for medical segmentation
• Zero-order optimization for layer-wise merging
• Single-task and multi-objective optimization regimes