UncertainSAM: Fast and Efficient Uncertainty Quantification of the Segment Anything Model

2025-05-08

AI Summary

To address the lack of uncertainty quantification (UQ) capability in the Segment Anything Model (SAM), this paper proposes USAM—a lightweight, plug-and-play post-training UQ method. USAM introduces the novel concept of *task uncertainty*, jointly modeling aleatoric, epistemic, and task-specific uncertainties via Bayesian entropy for unified quantification. It requires no architectural modification or retraining of SAM and produces uncertainty maps efficiently via a single deterministic forward pass. The method integrates prompt-aware uncertainty decomposition, post-training calibration, and lightweight CNN-based feature distillation. Evaluated on SA-V, MOSE, ADE20K, DAVIS, and COCO, USAM significantly outperforms existing UQ approaches while incurring negligible inference overhead. It natively supports user-provided prompts, semi-supervised learning, and flexible accuracy–efficiency trade-offs.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Uncertainty Representations
• Computer Vision: Segmentation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

The introduction of the Segment Anything Model (SAM) has paved the way for numerous semantic segmentation applications. For several tasks, quantifying the uncertainty of SAM is of particular interest. However, the ambiguous nature of the class-agnostic foundation model SAM challenges current uncertainty quantification (UQ) approaches. This paper presents a theoretically motivated uncertainty quantification model based on a Bayesian entropy formulation jointly respecting aleatoric, epistemic, and the newly introduced task uncertainty. We use this formulation to train USAM, a lightweight post-hoc UQ method. Our model traces the root of uncertainty back to under-parameterised models, insufficient prompts or image ambiguities. Our proposed deterministic USAM demonstrates superior predictive capabilities on the SA-V, MOSE, ADE20k, DAVIS, and COCO datasets, offering a computationally cheap and easy-to-use UQ alternative that can support user-prompting, enhance semi-supervised pipelines, or balance the tradeoff between accuracy and cost efficiency.

Problem

• Quantifying uncertainty in SAM for semantic segmentation tasks
• Addressing challenges in class-agnostic SAM uncertainty quantification
• Providing efficient uncertainty analysis to improve accuracy and cost efficiency

Innovation

• Bayesian entropy for uncertainty quantification
• Lightweight post-hoc UQ method USAM
• Deterministic USAM for efficient predictions