🤖 AI Summary
This work addresses the high cost of pixel-level annotations in medical image segmentation by proposing WeakMedSAM, a weakly supervised framework requiring only image-level or coarse-grained labels—no pixel-wise supervision. Methodologically, it introduces a novel subclass exploration module to mitigate interference from co-occurring lesions; a prompt affinity mining module that leverages SAM’s prompting capability to generate high-quality activation maps and guides random walk for precise segmentation; and integrates subclass contrastive learning, prompt-driven affinity graph modeling, and random-walk post-processing within a SAM-based architecture (compatible with SAMUS/EfficientSAM). Evaluated on BraTS 2019, AbdomenCT-1K, and MSD Cardiac benchmarks, WeakMedSAM significantly outperforms existing weakly supervised approaches and achieves performance comparable to state-of-the-art fully supervised methods.
📝 Abstract
We have witnessed remarkable progress in foundation models in vision tasks. Currently, several recent works have utilized the segmenting anything model (SAM) to boost the segmentation performance in medical images, where most of them focus on training an adaptor for fine-tuning a large amount of pixel-wise annotated medical images following a fully supervised manner. In this paper, to reduce the labeling cost, we investigate a novel weakly-supervised SAM-based segmentation model, namely WeakMedSAM. Specifically, our proposed WeakMedSAM contains two modules: 1) to mitigate severe co-occurrence in medical images, a sub-class exploration module is introduced to learn accurate feature representations. 2) to improve the quality of the class activation maps, our prompt affinity mining module utilizes the prompt capability of SAM to obtain an affinity map for random-walk refinement. Our method can be applied to any SAM-like backbone, and we conduct experiments with SAMUS and EfficientSAM. The experimental results on three popularly-used benchmark datasets, i.e., BraTS 2019, AbdomenCT-1K, and MSD Cardiac dataset, show the promising results of our proposed WeakMedSAM. Our code is available at https://github.com/wanghr64/WeakMedSAM.