roc analysis

Global AUC fails to reveal localized model performance deficiencies within subpopulations. To address this, we propose the first decomposable cluster-level AUC evaluation framework, which rigorously decomposes global AUC into two orthogonal components: intra-cluster ranking ability and inter-cluster discriminative ability. Methodologically, leveraging the geometric properties of ROC curves and cluster structure, we derive an additive, unbiased AUC decomposition formula and provide theoretical guarantees for its interpretability and statistical consistency. Unlike conventional metrics such as Brier score or log loss, our framework is the first to enable AUC decomposition at the cluster granularity, facilitating fine-grained diagnostic analysis and high-risk subgroup identification. Empirical evaluations on credit approval and fraud detection tasks demonstrate significant improvements in model validation accuracy and risk management efficacy.

This study addresses the challenge of effectively integrating radiologists’ assessments with AI predictions in mammographic screening to optimize rule-out and rule-in diagnostic strategies. It introduces, for the first time, a unified joint ROC theoretical framework tailored to both clinical scenarios. By modeling the dependence between physician and AI diagnostic outputs using bivariate copulas, the work theoretically derives—and empirically validates—the impact of their correlation on AUC performance: higher correlation improves rule-out efficacy in diseased populations, whereas lower correlation is preferable in non-diseased populations; conversely, for rule-in tasks, the opposite pattern holds. This framework provides a rigorous theoretical foundation and practical guidance for designing collaborative diagnostic systems that strategically leverage human–AI synergy.

This work investigates the geometric foundations of ROC and PR curves in binary classification, aiming to unify the understanding of curve morphology and classifier behavior through a geometric lens. Methodologically, it introduces the composite function (G = F_p circ F_n^{-1}) as a core modeling framework—where (F_p) and (F_n) denote the CDFs of positive and negative class score distributions—and rigorously establishes a geometric mapping between ROC/PR curve shapes and the underlying distributional geometry. It reveals that (G) quantifies inter-class leakage and admits interpretation via KL divergence. Furthermore, it derives geometric criteria for classifier dominance and interpretability grounded in differential geometry, statistical inference, and CDF transformation theory. The contributions include: (i) a principled, geometrically interpretable framework for threshold selection; (ii) robust, distribution-agnostic tools for classifier comparison; and (iii) enhanced reliability and adaptability in cost-sensitive deployment—particularly under class imbalance and distributional overlap.

This study addresses the overreliance on the Area Under the ROC Curve (AUC) in software defect prediction research, which can lead to biased model evaluation as AUC fails to capture a model’s discriminative performance across all classification thresholds. To overcome this limitation, the authors propose an augmented ROC curve and alternative visualization techniques that explicitly model the true positive rate and false positive rate as functions of the decision threshold, annotating corresponding threshold points directly on the ROC curve. Their analysis demonstrates that a high AUC does not guarantee superior performance over random guessing at every threshold, thereby exposing critical shortcomings of conventional evaluation practices. The work underscores the necessity of incorporating multi-threshold classification performance into a more comprehensive and nuanced assessment framework.

This study investigates how correlations among biomarkers influence the discriminative performance of predictive models, elucidating the mechanism by which adding new biomarkers does not necessarily improve model accuracy. Through theoretical derivations under multivariate normal and skewed distributions, simulation experiments—including log-folded bivariate normal and Gamma distributions—and validation using serum metabolomic data from pancreatic ductal adenocarcinoma patients, the work establishes, for the first time, an analytical relationship between biomarker correlation structures and the area under the ROC curve (AUC). The findings demonstrate that negative correlation most substantially enhances the joint AUC when individual biomarkers exhibit comparable predictive power, and real-world metabolomic data confirm that inter-biomarker correlation plays a decisive role in the performance of disease detection models.

This study addresses the lack of decision-oriented evaluation methodologies in current machine translation quality estimation (QE) systems. It introduces receiver operating characteristic (ROC) analysis into QE evaluation for the first time, complementing and validating against conventional metrics. Experimental results demonstrate that ROC analysis not only aligns consistently with existing evaluation outcomes but also yields actionable performance insights. By providing a clearer understanding of trade-offs between true positive and false positive rates across varying decision thresholds, this approach significantly enhances the practical utility of QE assessment and offers robust guidance for deployment decisions in real-world applications.

Current evaluation practices for supervised learning models are often misleading due to an overreliance on single aggregate metrics, which neglect the alignment among data characteristics, task objectives, and real-world application contexts. This work reframes model evaluation as a context-dependent, decision-oriented process and systematically investigates—through controlled experiments—the impact of dataset properties, validation strategies, class imbalance, and asymmetric error costs on evaluation outcomes. Leveraging diverse benchmark datasets, multiple validation protocols, and multidimensional performance measures, the study uncovers common pitfalls such as the accuracy paradox, data leakage, and metric misuse. It proposes a structured evaluation framework explicitly aligned with operational goals, offering principled guidance for developing more robust, reliable, and trustworthy supervised learning systems.

This work addresses the long-standing challenge of model selection in anomaly detection under scenarios lacking labeled validation data, where existing approaches rely heavily on reconstruction quality or manual annotations. The authors propose AUCp, a fully unsupervised model selection metric that treats the entire unlabeled test set as positive (i.e., anomalous) and computes the Area Under the ROC Curve (AUC) accordingly. This approach enables, for the first time, completely annotation-free model selection across diverse unsupervised and self-supervised anomaly detection frameworks. Evaluated on medical image reconstruction tasks involving multiple neurological disorders and heterogeneous datasets, AUCp consistently outperforms conventional metrics, effectively identifying the best-performing inference models and thereby enhancing overall anomaly detection performance.

This study addresses the evaluation and selection of source-level likelihood ratio (LR) systems for forensic evidence-to-reference comparison tasks by proposing an integrated analytical framework that balances performance and practical feasibility. The authors employ strictly proper scoring rules to quantify how effectively each system updates Bayesian prior odds and present the first systematic comparison among specific-source feature-based, common-source anchored, and unanchored score-based LR approaches. Their findings reveal that specific-source feature-based LRs achieve the highest performance but incur substantial experimental costs, whereas common-source feature-based methods offer strong discriminative power with significantly reduced implementation complexity. All LR systems substantially outperform a baseline relying solely on prior odds. This work thus provides both theoretical grounding and practical guidance for selecting LR systems in forensic practice.

This study addresses the limitations of evaluating multiclass classifiers using single performance metrics, which often leads to misleading conclusions. To overcome this, the work proposes a multidimensional evaluation paradigm that leverages the PyCM library to construct a comprehensive analytical framework, enabling systematic comparison of classifier performance across a diverse set of evaluation metrics. Through two case studies, the research uncovers nuanced performance trade-offs that conventional metrics fail to capture, thereby demonstrating the necessity and effectiveness of multidimensional assessment in model selection and optimization. The findings further highlight the unique value of PyCM in facilitating thorough and precise evaluation of multiclass classification systems.