Learning from Noisy Prompts: Saliency-Guided Prompt Distillation for Robust Segmentation with SAM

2026-04-25

AI Summary

This work addresses the significant performance degradation of the Segment Anything Model (SAM) when applied to clinical medical images with commonly used but coarse, ambiguous, and noisy prompts—such as centerline points. To overcome this limitation, the authors propose the SPD framework, which employs a lightweight saliency head to learn anatomical priors and generate reliable localization maps. SPD further enhances noisy prompts by fusing contextual information from adjacent slices to validate and refine them into a consensus prompt set, while enforcing slice-pair consistency constraints to improve local anatomical coherence. Notably, SPD is the first method to integrate saliency guidance with cross-slice context distillation, transforming unreliable clinical prompts into robust supervisory signals that mimic expert reasoning, thereby enabling SAM deployment without fine-grained annotations. Experiments on four MRI/CT datasets demonstrate that SPD consistently outperforms existing SAM adaptations and supervised baselines in both regional and boundary segmentation metrics.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Segmentation is central to clinical diagnosis and monitoring, yet the reliability of modern foundation models in medical imaging still depends on the availability of precise prompts. The Segment Anything Model (SAM) offers powerful zero-shot capabilities, although it collapses under the weak, generic, and noisy prompts that dominate real clinical workflows. In practice, annotations such as centerline points are coarse and ambiguous, often drifting across neighboring anatomy and misguiding SAM toward inconsistent or incomplete masks. We introduce SPD, a Saliency-Guided Prompt Distillation framework that converts these unreliable cues into robust guidance. SPD first learns data-driven anatomical priors through a lightweight saliency head to obtain confident localization maps. These priors then drive Contextual Prompt Distillation, which validates and enriches noisy prompts using cues from anatomically adjacent slices, producing a consensus prompt set that matches the behavior of expert reasoning. A Pairwise Slice Consistency objective further enforces local anatomical coherence during segmentation. Experiments on four challenging MRI and CT benchmarks demonstrate that SPD consistently outperforms existing SAM adaptations and supervised baselines, delivering large gains in both region-based and boundary-based metrics. SPD provides a practical and principled path toward reliable foundation model deployment in clinical environments where only imperfect prompts are available.

Problem

• noisy prompts
• medical image segmentation
• Segment Anything Model
• clinical workflows
• prompt reliability

Innovation

• Prompt Distillation
• Saliency Guidance
• Slice Consistency
• Robust Segmentation
• Foundation Models