Concept Alignment Contrast and Long-Short Prompt Memory for Test-Time Adaptation of SAM3 in Medical Image Segmentation

2026-06-22

AI Summary

This work addresses the performance degradation of SAM3 in medical image segmentation under domain shift, where existing test-time adaptation methods struggle to simultaneously ensure semantic correctness and adaptation stability. To tackle this challenge, the authors propose CM-TTA, a novel framework that introduces a concept-aligned contrastive metric to evaluate semantic consistency across multi-view augmentations and selects high-quality predictions as supervision signals. Furthermore, CM-TTA incorporates a long-short prompt memory module with a dense supervision prompting strategy, effectively balancing local responsiveness and global stability during adaptation. Extensive experiments on prostate and skin lesion segmentation demonstrate that CM-TTA significantly outperforms current SAM3 adaptation approaches, achieving substantial improvements in cross-domain segmentation performance.

Abstract

Concept segmentation models like Segment Anything Model 3 (SAM3) show strong generalization on natural images, yet their performance degrades in medical imaging due to the domain gap caused by different imaging principles and styles. Test-Time Adaptation (TTA) is essential for improving the testing performance by updating the model on the fly without annotations. However, existing vision-language TTA methods are mainly driven by image-level uncertainty minimization, which does not necessarily reflect region-level semantic correctness in medical segmentation. Moreover, they often lack mechanisms to maintain stability in continual one-pass adaptation, leading to limited performance when reliable dense supervision is missing for segmentation. To address these issues, we propose Concept Alignment Contrast and LongShort Prompt Memory for Test-Time Adaptation (CM-TTA) of SAM3 for medical images. First, for a test sample with multiple augmentations, we introduce a novel Concept Alignment Contrast (CAC) metric, which leverages textual-visual semantic consistency to robustly evaluate prediction quality to select the best augmented view as the supervision. Second, to balance rapid and stable adaptation, we design a Long-Short Prompt Memory (LSPM) module. The short memory dynamically fuses recent prompts based on CAC scores for agile local adaptation, while the long memory maintains a stable global prompt to generate enhanced pseudo-labels. Finally, a Densely Supervised Prompt Update (DSPU) strategy is proposed to optimize the prompt embeddings with enhanced pseudo labels as dense supervision. Extensive experiments on prostate and skin lesion segmentation demonstrate that our CM-TTA framework significantly outperforms existing methods for TTA of SAM3.

Problem

• Test-Time Adaptation
• Medical Image Segmentation
• Domain Gap
• Semantic Consistency
• Dense Supervision

Innovation

• Concept Alignment Contrast
• Long-Short Prompt Memory
• Test-Time Adaptation
• Medical Image Segmentation
• SAM3