Dual-Adaptive SAM3: Hierarchical Routing over Low-Rank Expert Layers for Parameter-Efficient Medical Image Segmentation

2026-06-30

AI Summary

This work addresses the inefficiency of full-parameter fine-tuning of large vision-language models like SAM3 and the high computational cost of standard Mixture-of-Experts (MoE) approaches in medical image segmentation. To this end, the authors propose a dual adaptive efficient adaptation mechanism: first, a Dynamic Expert Router (DER) enables task-aware sparse expert activation through vision-text joint reasoning; second, Decomposed Parameterized Experts (DPE) introduce lightweight low-rank incremental parameters trained while keeping the SAM3 backbone frozen. The proposed method achieves approximately 5% performance gains over current state-of-the-art approaches on multiple public medical segmentation benchmarks, matching the accuracy of fully fine-tuned SAM3 while reducing trainable parameters by over 80%, thereby offering high accuracy, efficiency, and interpretability.

Abstract

The Segment Anything Model with Concepts (SAM3) heralds a new paradigm for open-vocabulary segmentation through natural language interaction, offering significant potential for medical image analysis. However, effectively adapting such a powerful vision-language model to the diverse and nuanced domain of medical imaging remains a key challenge. Naive fine-tuning is parameter-inefficient, while standard Mixture-of-Experts (MoE) methods introduce prohibitive computational overhead, limiting their clinical applicability. To address this, we propose Dual-Adaptive SAM3 (DA-SAM3), a novel framework that achieves both high segmentation accuracy and extreme parameter efficiency via a dual-adaptive specialization mechanism. Our first adaptation is task-aware: a Dynamic Expert Router (DER) that sparsely activates the most relevant experts by jointly reasoning about the visual input and the textual concept prompt, mimicking a clinical consultation process. Our second adaptation is parameter-aware: a Decomposed Parameterized Experts (DPE) design that represents each expert as a shared frozen base (inherited from the pretrained SAM3) and a lightweight trainable low-rank delta, reducing MoE parameter overhead by over 80\%. Extensive experiments on multiple public medical segmentation benchmarks demonstrate that Dual-Adaptive SAM3 not only matches or exceeds the accuracy of fully fine-tuned SAM3 and standard MoE baselines, but also achieves a notable 5\% gain over current state-of-the-art methods, with interpretable results validating its effectiveness. The code is available at: https://github.com/Reconsider80/DA-SAM3.

Problem

• medical image segmentation
• parameter efficiency
• vision-language model
• Mixture-of-Experts
• model adaptation

Innovation

• Dynamic Expert Router
• Decomposed Parameterized Experts
• Low-Rank Adaptation
• Mixture-of-Experts
• Parameter-Efficient Tuning