From Adaptation to Generalization: Adaptive Visual Prompting for Medical Image Segmentation

2026-04-19

AI Summary

Existing vision prompting methods struggle to handle both intra-domain and inter-domain variability in medical image segmentation, leading to limited generalization. This work proposes APEX, a framework that dynamically generates adaptive prompts for each input through a learnable prompt memory bank and incorporates a Fourier spectrum–based domain feature querying mechanism. To further enhance prompt discriminability and robustness, the method introduces Low-frequency Feature Contrastive learning (LFC), which effectively aggregates intra-domain features while separating inter-domain ones. Designed as a plug-and-play module, APEX is compatible with various backbone networks and demonstrates significant improvements in generalization performance across both seen and unseen domains on two medical segmentation benchmarks.

Technology Category

• Computer Vision: Segmentation
• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Visual prompting has emerged as a powerful method for adapting pre-trained models to new domains without updating model parameters. However, existing prompting methods typically optimize a single prompt per domain and apply it uniformly to all inputs, limiting their ability to generalize under intra and inter-domain variability, which is especially critical in the medical field. To address this, we propose APEX, an Adaptive Prompt EXtraction framework that retrieves input-specific prompts from a learnable prompt memory. The memory stores diverse, domain-discriminative prompt representations and is queried via domain features extracted from the Fourier spectrum. To learn robust and discriminative domain features, we introduce a novel Low-Frequency Feature Contrastive (LFC) learning framework that clusters representations from the same domain while separating those from different domains. Extensive experiments on two medical segmentation tasks demonstrate that APEX significantly improves generalization across both seen and unseen domains. Furthermore, it complements any existing backbones and consistently enhances performance, confirming its effectiveness as a plug-and-play prompting solution in medical fields. The code is available at https://github.com/cetinkayaevren/apex/

Problem

• visual prompting
• medical image segmentation
• domain generalization
• intra-domain variability
• inter-domain variability

Innovation

• adaptive visual prompting
• prompt memory
• Fourier spectrum
• low-frequency feature contrastive learning
• domain generalization