Weakly Supervised Pixel-Level Annotation with Visual Interpretability

2025-02-25

AI Summary

Medical image pixel-level annotation relies heavily on expert knowledge, leading to high time/cost overhead and inter-annotator inconsistency. To address this, we propose a weakly supervised, interpretable segmentation framework that requires only image-level labels. Our method integrates ResNet50, EfficientNet, and DenseNet ensembles; generates saliency maps via XGrad-CAM; and quantifies predictive uncertainty using Monte Carlo Dropout. Crucially, we pioneer the joint modeling of visual interpretability and uncertainty—leveraging intersection-based aggregation of multi-model saliency maps to enable robust, pixel-free decision-making and automatically flag low-confidence regions for human review. Evaluated on TBX11K and Fire datasets, our approach achieves classification accuracies of 93.04% and 96.4%, and pixel-wise IoU scores of 36.07% and 64.7%, respectively—substantially outperforming fully supervised baselines while ensuring high accuracy, transparency, and clinical trustworthiness.

Technology Category

• Computer Vision: Interpretability, Explainability, and Transparency
• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Uncertainty Representations

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Medical image annotation is essential for diagnosing diseases, yet manual annotation is time-consuming, costly, and prone to variability among experts. To address these challenges, we propose an automated explainable annotation system that integrates ensemble learning, visual explainability, and uncertainty quantification. Our approach combines three pre-trained deep learning models - ResNet50, EfficientNet, and DenseNet - enhanced with XGrad-CAM for visual explanations and Monte Carlo Dropout for uncertainty quantification. This ensemble mimics the consensus of multiple radiologists by intersecting saliency maps from models that agree on the diagnosis while uncertain predictions are flagged for human review. We evaluated our system using the TBX11K medical imaging dataset and a Fire segmentation dataset, demonstrating its robustness across different domains. Experimental results show that our method outperforms baseline models, achieving 93.04% accuracy on TBX11K and 96.4% accuracy on the Fire dataset. Moreover, our model produces precise pixel-level annotations despite being trained with only image-level labels, achieving Intersection over Union IoU scores of 36.07% and 64.7%, respectively. By enhancing the accuracy and interpretability of image annotations, our approach offers a reliable and transparent solution for medical diagnostics and other image analysis tasks.

Problem

• Automates medical image annotation process
• Enhances accuracy with ensemble learning models
• Provides visual explainability and uncertainty quantification

Innovation

• Ensemble learning integrates multiple models
• XGrad-CAM provides visual explainability
• Monte Carlo Dropout quantifies uncertainty