CONSIGN: Conformal Segmentation Informed by Spatial Groupings via Decomposition

2025-05-20

AI Summary

Existing image segmentation models lack statistically rigorous pixel-wise confidence estimates; directly applying conformal prediction (CP) ignores spatial correlations, yielding overly conservative and uninterpretable uncertainty quantification. To address this, we propose Spatial-Conformal Prediction (SCP), the first CP framework incorporating a spatial clustering decomposition mechanism that explicitly relaxes the pixel independence assumption, enabling structured and well-calibrated uncertainty estimation. SCP is model-agnostic—compatible with any pre-trained segmentation architecture—and supports diverse uncertainty modeling strategies, including dropout, ensemble, and Bayesian inference. Evaluated on three medical imaging datasets and two COCO subsets, SCP yields significantly more compact prediction sets and improved coverage reliability compared to standard pixel-wise CP. It achieves substantial gains across multiple uncertainty evaluation metrics—including calibration error, average set size, and empirical coverage—while preserving theoretical validity and enhancing interpretability through spatially coherent uncertainty maps.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Segmentation
• Reasoning under Uncertainty: Uncertainty Representations

Application Category

• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Most machine learning-based image segmentation models produce pixel-wise confidence scores - typically derived from softmax outputs - that represent the model's predicted probability for each class label at every pixel. While this information can be particularly valuable in high-stakes domains such as medical imaging, these (uncalibrated) scores are heuristic in nature and do not constitute rigorous quantitative uncertainty estimates. Conformal prediction (CP) provides a principled framework for transforming heuristic confidence scores into statistically valid uncertainty estimates. However, applying CP directly to image segmentation ignores the spatial correlations between pixels, a fundamental characteristic of image data. This can result in overly conservative and less interpretable uncertainty estimates. To address this, we propose CONSIGN (Conformal Segmentation Informed by Spatial Groupings via Decomposition), a CP-based method that incorporates spatial correlations to improve uncertainty quantification in image segmentation. Our method generates meaningful prediction sets that come with user-specified, high-probability error guarantees. It is compatible with any pre-trained segmentation model capable of generating multiple sample outputs - such as those using dropout, Bayesian modeling, or ensembles. We evaluate CONSIGN against a standard pixel-wise CP approach across three medical imaging datasets and two COCO dataset subsets, using three different pre-trained segmentation models. Results demonstrate that accounting for spatial structure significantly improves performance across multiple metrics and enhances the quality of uncertainty estimates.

Problem

• Transforming heuristic confidence scores into valid uncertainty estimates
• Addressing spatial correlations in image segmentation for better uncertainty quantification
• Generating prediction sets with high-probability error guarantees for segmentation models

Innovation

• Incorporates spatial correlations for better uncertainty
• Uses conformal prediction for statistically valid estimates
• Compatible with various pre-trained segmentation models