Uncertainty-Aware Segmentation Quality Prediction via Deep Learning Bayesian Modeling: Comprehensive Evaluation and Interpretation on Skin Cancer and Liver Segmentation

2025-08-02

AI Summary

To address unreliable quality assessment in clinical medical image segmentation caused by the absence of ground-truth annotations, this paper proposes two unsupervised segmentation quality prediction frameworks. Methodologically, we integrate multi-source uncertainty modeling—including Monte Carlo Dropout, ensemble learning, and test-time augmentation—with deep architectures (SwinUNet and FPN-ResNet50), aggregating uncertainty metrics such as entropy and mutual information, and enhancing interpretability via visualization. Our key contribution is a novel multi-uncertainty aggregation strategy that significantly improves cross-modal robustness. Evaluation on the HAM10000 skin lesion dataset yields an R² of 93.25% and a Pearson correlation coefficient of 96.58%; for 3D liver segmentation, R² reaches 85.03%. These results demonstrate the method’s effectiveness, generalizability, and clinical applicability.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Uncertainty Representations

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Image segmentation is a critical step in computational biomedical image analysis, typically evaluated using metrics like the Dice coefficient during training and validation. However, in clinical settings without manual annotations, assessing segmentation quality becomes challenging, and models lacking reliability indicators face adoption barriers. To address this gap, we propose a novel framework for predicting segmentation quality without requiring ground truth annotations during test time. Our approach introduces two complementary frameworks: one leveraging predicted segmentation and uncertainty maps, and another integrating the original input image, uncertainty maps, and predicted segmentation maps. We present Bayesian adaptations of two benchmark segmentation models-SwinUNet and Feature Pyramid Network with ResNet50-using Monte Carlo Dropout, Ensemble, and Test Time Augmentation to quantify uncertainty. We evaluate four uncertainty estimates: confidence map, entropy, mutual information, and expected pairwise Kullback-Leibler divergence on 2D skin lesion and 3D liver segmentation datasets, analyzing their correlation with segmentation quality metrics. Our framework achieves an R2 score of 93.25 and Pearson correlation of 96.58 on the HAM10000 dataset, outperforming previous segmentation quality assessment methods. For 3D liver segmentation, Test Time Augmentation with entropy achieves an R2 score of 85.03 and a Pearson correlation of 65.02, demonstrating cross-modality robustness. Additionally, we propose an aggregation strategy that combines multiple uncertainty estimates into a single score per image, offering a more robust and comprehensive assessment of segmentation quality. Finally, we use Grad-CAM and UMAP-based embedding analysis to interpret the model's behavior and reliability, highlighting the impact of uncertainty integration.

Problem

• Predicts segmentation quality without ground truth annotations
• Evaluates uncertainty-aware models for clinical adoption
• Combines multiple uncertainty estimates for robust assessment

Innovation

• Bayesian deep learning for uncertainty-aware segmentation
• Monte Carlo Dropout and Test Time Augmentation
• Aggregation of multiple uncertainty estimates per image