A theory of ROC analysis of rule-out and rule-in diagnostics with applications to mammography data

2026-04-25

AI Summary

This study addresses the challenge of effectively integrating radiologists’ assessments with AI predictions in mammographic screening to optimize rule-out and rule-in diagnostic strategies. It introduces, for the first time, a unified joint ROC theoretical framework tailored to both clinical scenarios. By modeling the dependence between physician and AI diagnostic outputs using bivariate copulas, the work theoretically derives—and empirically validates—the impact of their correlation on AUC performance: higher correlation improves rule-out efficacy in diseased populations, whereas lower correlation is preferable in non-diseased populations; conversely, for rule-in tasks, the opposite pattern holds. This framework provides a rigorous theoretical foundation and practical guidance for designing collaborative diagnostic systems that strategically leverage human–AI synergy.

Technology Category

• Humans and AI: Planning and Decision Support for Human-Machine Teams
• Machine Learning: Calibration & Uncertainty Quantification
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Multiple diagnostic tests are frequently used to determine the presence of a disease condition in patients. In this paper, we use bivariate copulas to examine the properties of receiver operating characteristic (ROC) curves formed when two correlated diagnostic tests are used together to rule-out ("believe the negative") and rule-in ("believe the positive") patients for disease. We use this theory to analyze three mammography data sets where AI devices are applied to reduce radiologists' workload or improve diagnostic performance. Our analysis shows with generality that increasing the radiologist-AI correlation for diseased cases enhances the area under the ROC curve (AUC) of a radiologist-AI rule-out curve, whereas decreasing correlation for non-diseased cases has a similar effect. The opposite trends hold for rule-in scenarios. Applications to clinical mammography data show that projected empirical radiologist performance under a rule-out or rule-in scenario is consistent with the theory.

Problem

• ROC analysis
• rule-out
• rule-in
• diagnostic tests
• mammography

Innovation

• bivariate copulas
• ROC analysis
• rule-out/rule-in diagnostics
• radiologist-AI correlation
• mammography