SAIF: A Stability-Aware Inference Framework for Medical Image Segmentation with Segment Anything Model

2026-03-13

AI Summary

This work addresses the inference instability of the Segment Anything Model (SAM) in medical image segmentation, which arises from localization errors in bounding box prompts and fixed-threshold binarization. The authors propose a training-free, plug-and-play inference framework that explicitly models the joint uncertainty of prompts and thresholds during inference for the first time. By applying structured perturbations to bounding boxes and sampling multiple thresholds, the method generates a set of diverse predictions. These predictions are then fused via a novel scoring mechanism that evaluates decision stability and boundary consistency to assign adaptive weights. Evaluated on multiple medical imaging benchmarks—including Synapse, CVC-ClinicDB, Kvasir-SEG, and CVC-300—the approach significantly improves both segmentation accuracy and boundary robustness without requiring any architectural modifications or retraining of the base model.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Agentic search
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Segment Anything Model (SAM) enable scalable medical image segmentation but suffer from inference-time instability when deployed as a frozen backbone. In practice, bounding-box prompts often contain localization errors, and fixed threshold binarization introduces additional decision uncertainty. These factors jointly cause high prediction variance, especially near object boundaries, degrading reliability. We propose the Stability-Aware Inference Framework (SAIF), a training-free and plug-and-play inference framework that improves robustness by explicitly modeling prompt and threshold uncertainty. SAIF constructs a joint uncertainty space via structured box perturbations and threshold variations, evaluates each hypothesis using decision stability and boundary consistency, and introduces a stability-consistency score to filter unstable candidates and perform stability-weighted fusion in probability space. Experiments on Synapse, CVC-ClinicDB, Kvasir-SEG, and CVC-300 demonstrate that SAIF consistently improves segmentation accuracy and robustness, achieving state-of-the-art performance without retraining or architectural modification. Our anonymous code is released at https://anonymous.4open.science/r/SAIF.

Problem

• medical image segmentation
• inference instability
• prompt uncertainty
• threshold binarization
• prediction variance

Innovation

• Stability-Aware Inference
• Uncertainty Modeling
• Prompt Perturbation
• Threshold Variation
• Boundary Consistency