Part-aware Personalized Segment Anything Model for Patient-Specific Segmentation

2024-03-08

arXiv.org

AI Summary

To address the challenge of adapting medical image segmentation to substantial inter-patient anatomical variability and scarce per-patient annotated data in precision medicine, this paper proposes a fine-tuning-free, single-sample patient-adaptive segmentation method. Built upon the Segment Anything Model (SAM), our approach introduces a part-aware prompting mechanism and an anomaly-aware prompt retrieval strategy. Specifically, it leverages part-level feature guidance for multi-point prompt selection, employs robust prompt retrieval, and integrates zero-shot transfer techniques—enabling personalized segmentation from just one patient sample (e.g., a single image plus a few point prompts). To our knowledge, this is the first work achieving true fine-tuning-free, single-sample patient adaptation in medical image segmentation. Evaluated on two patient-specific segmentation tasks, our method achieves average Dice score improvements of +8.0% and +2.0%, respectively, and boosts mIoU by +6.4% on the PerSeg benchmark.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Precision medicine, such as patient-adaptive treatments utilizing medical images, poses new challenges for image segmentation algorithms due to (1) the large variability across different patients and (2) the limited availability of annotated data for each patient. In this work, we propose a data-efficient segmentation method to address these challenges, namely Part-aware Personalized Segment Anything Model (P^2SAM). Without any model fine-tuning, P^2SAM enables seamless adaptation to any new patients relying only on one-shot patient-specific data. We introduce a novel part-aware prompt mechanism to select multiple-point prompts based on part-level features of the one-shot data. To further promote the robustness of the selected prompt, we propose a retrieval approach to handle outlier prompts. Extensive experiments demonstrate that P^2SAM improves the performance by +8.0% and +2.0% mean Dice score within two patient-specific segmentation settings, and exhibits impressive generality across different application domains, e.g., +6.4% mIoU on the PerSeg benchmark. Code will be released upon acceptance.

Problem

• Adapting segmentation algorithms to new patients with limited annotated data
• Enhancing segmentation accuracy using part-aware prompts without model fine-tuning
• Improving robustness in patient-adaptive and natural image segmentation tasks

Innovation

• Part-aware prompt mechanism for feature selection
• Distribution-guided retrieval for optimal part number
• One-shot patient-specific data adaptation