A Unified Inference Method for FROC-type Curves and Related Summary Indices

2025-12-23

AI Summary

This paper addresses the lack of unified statistical inference for Free-response Receiver Operating Characteristic (FROC) curves and their summary metrics—such as area under the curve (AUC) and sensitivity at fixed false-positive rates—in multi-object detection and localization tasks. We propose the first joint asymptotic inference framework grounded in the initial-detection–candidate-modelling paradigm. Using maximum likelihood estimation, we simultaneously fit a smooth FROC curve and construct confidence intervals for key performance metrics. We establish the asymptotic normality of the estimators, overcoming limitations of conventional nonparametric or heuristic approaches. Monte Carlo simulations demonstrate excellent finite-sample properties. The method is successfully applied to SaMD-based pulmonary lesion diagnosis assessment, delivering interpretable, reproducible, and statistically rigorous inference results.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Probabilistic Inference
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

Free-response observer performance studies are of great importance for accuracy evaluation and comparison in tasks related to the detection and localization of multiple targets or signals. The free-response receiver operating characteristic (FROC) curve and many similar curves based on the free-response observer performance assessment data are important tools to display the accuracy of detection under different thresholds. The true positive rate at a fixed false positive rate and summary indices such as the area under the FROC curve are also commonly used as the figures of merit in the statistical evaluation of these studies. Motivated by a free-response observer performance assessment research of a Software as a Medical Device (SaMD), we propose a unified method based on the initial-detection-and-candidate model to simultaneously estimate a smooth curve and derive confidence intervals for summary indices and the true positive rate at a fixed false positive rate. A maximum likelihood estimator is proposed and its asymptotic normality property is derived. Confidence intervals are constructed based on the asymptotic normality of our maximum likelihood estimator. Simulation studies are conducted to evaluate the finite sample performance of the proposed method. We apply the proposed method to evaluate the diagnostic performance of the SaMD for detecting pulmonary lesions.

Problem

• Unified inference for FROC curves and summary indices
• Estimates smooth curves and confidence intervals simultaneously
• Evaluates diagnostic performance of medical software for lesion detection

Innovation

• Unified method for FROC curve estimation
• Maximum likelihood estimator with asymptotic normality
• Confidence intervals for summary indices and TPR