Conformal Risk Control for Pulmonary Nodule Detection

2024-12-28

AI Summary

Lung nodule detection faces challenges including high model uncertainty, substantial inter-rater disagreement among radiologists, and a high false-positive rate. Method: This paper introduces Conformal Risk Control (CRC) to lung cancer screening for the first time, integrating state-of-the-art object detection with conformal prediction. We propose a risk-set construction strategy that quantifies epistemic uncertainty—arising from multi-expert annotation discrepancies—and enables adjustable trade-offs between sensitivity and false-positive rate. Results: Evaluated on consensus annotations from three radiologists, our framework achieves sensitivity comparable to that of a single expert while rigorously satisfying user-specified statistical validity guarantees (e.g., 90% coverage probability). This work delivers clinically deployable, interpretable, and risk-controlled AI predictions, and establishes a novel paradigm for uncertainty modeling via conformal inference in medical imaging.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Uncertainty Representations
• Computer Vision: Interpretability, Explainability, and Transparency

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Quantitative tools are increasingly appealing for decision support in healthcare, driven by the growing capabilities of advanced AI systems. However, understanding the predictive uncertainties surrounding a tool's output is crucial for decision-makers to ensure reliable and transparent decisions. In this paper, we present a case study on pulmonary nodule detection for lung cancer screening, enhancing an advanced detection model with an uncertainty quantification technique called conformal risk control (CRC). We demonstrate that prediction sets with conformal guarantees are attractive measures of predictive uncertainty in the safety-critical healthcare domain, allowing end-users to achieve arbitrary validity by trading off false positives and providing formal statistical guarantees on model performance. Among ground-truth nodules annotated by at least three radiologists, our model achieves a sensitivity that is competitive with that generally achieved by individual radiologists, with a slight increase in false positives. Furthermore, we illustrate the risks of using off-the-shelve prediction models when faced with ontological uncertainty, such as when radiologists disagree on what constitutes the ground truth on pulmonary nodules.

Problem

• Lung Nodule Detection
• Model Uncertainty
• Inter-observer Variability

Innovation

• Consistent Risk Control
• Lung Nodule Detection
• Uncertainty Quantification