On The Robustness of Foundational 3D Medical Image Segmentation Models Against Imprecise Visual Prompts

2026-01-23

AI Summary

Current foundation models for 3D medical image segmentation exhibit insufficient robustness when confronted with imprecise visual prompts. This study presents the first systematic evaluation of two state-of-the-art 3D foundation models on multi-organ abdominal segmentation, specifically assessing their sensitivity to controlled perturbations in visual prompts that simulate real-world deviations in shape and spatial location. The findings reveal that while these models heavily rely on such prompt-derived cues, they also demonstrate a degree of resilience to certain types of perturbations. By delineating the robustness boundaries of prompt-driven segmentation under clinically plausible conditions, this work provides critical insights for improving the reliability of such methods in practical clinical deployment.

Technology Category

• Computer Vision: Segmentation
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Machine Learning: Deep Neural Architectures and Foundation Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

While 3D foundational models have shown promise for promptable segmentation of medical volumes, their robustness to imprecise prompts remains under-explored. In this work, we aim to address this gap by systematically studying the effect of various controlled perturbations of dense visual prompts, that closely mimic real-world imprecision. By conducting experiments with two recent foundational models on a multi-organ abdominal segmentation task, we reveal several facets of promptable medical segmentation, especially pertaining to reliance on visual shape and spatial cues, and the extent of resilience of models towards certain perturbations. Codes are available at: https://github.com/ucsdbiag/Prompt-Robustness-MedSegFMs

Problem

• 3D medical image segmentation
• foundational models
• visual prompts
• robustness
• imprecise prompts

Innovation

• 3D foundational models
• medical image segmentation
• visual prompts
• prompt robustness
• controlled perturbations