Value-of-Information Analysis for External Validation of Risk Prediction Models in Multicenter Studies and Systematic Reviews

2026-07-02

AI Summary

This study addresses the challenge of evaluating whether risk prediction models outperform default strategies in external validation settings, where limited sample sizes and unaccounted multicenter heterogeneity often hinder reliable assessment. The authors extend the expected value of perfect information (EVPI) and expected value of partial perfect information (EVPPI) to a multicenter and systematic review framework, introducing novel metrics—EVPIglobal, EVPIcluster_j, EVPIcluster, and EVPPIcluster,prevalence—to distinguish between global and local optimal strategies and to separate observed from unobserved centers, thereby disentangling sources of heterogeneity. Leveraging the MetaNB R package within a Bayesian decision-theoretic and meta-analytic framework, they conduct a value-of-information analysis of the ADNEX model across 36 centers. Results indicate that adopting ADNEX globally requires no additional data (EVPIglobal = 0); eliminating uncertainty about center-specific performance and prevalence could reduce false positives by 1,134 cases annually, with prevalence uncertainty alone accounting for 158 avoidable false positives; and in unobserved centers, there remains a 3% probability that the default strategy is superior.

Abstract

External validation studies have finite sample sizes, creating uncertainty about whether a prediction model's Net Benefit (NB) exceeds default strategies' NB. The expected value of perfect information (EVPI) quantifies consequences of uncertainty. Current EVPI methods focus on single studies, ignoring between-center heterogeneity. We extend EVPI and expected value of partial perfect information (EVPPI) to account for between-cluster heterogeneity in multicenter studies and meta-analyses. We distinguish between the global and local optimal strategy and between observed and unobserved clusters. We define EVPIglobal, EVPIcluster_j, EVPIcluster, and EVPPIcluster,prevalence, implemented in the MetaNB R package, and illustrate them using a systematic review across 36 centers of the ADNEX model for ovarian cancer diagnosis. Assuming one global decision regarding ADNEX adoption, there is no need for further data to confirm ADNEX is superior overall (EVPIglobal 0). Meta-analysis borrows information across observed clusters, resulting in consistent local superiority of ADNEX and nonzero but typically lower EVPIcluster_j than when considering local data alone. There is 0.03 probability default strategies are superior in unobserved centers. Eliminating uncertainty on performance and prevalence in each (EVPIcluster) would gain 1134 net avoided false positives (FP) per year, assuming 350000 tumors annually with 20% malignancies. Determining only local prevalence with certainty (EVPPIcluster, prevalence) would gain net 158 avoided FP per year. EVPI extensions disentangle sources of uncertainty and quantify the need for further validation to determine the global or locally optimal strategy. Considering uncertainty and heterogeneity in clinical utility across clusters is essential to decide whether additional validation studies are warranted.

Problem

• value of information
• external validation
• risk prediction models
• between-center heterogeneity
• net benefit

Innovation

• value of information
• external validation
• between-center heterogeneity
• net benefit
• meta-analysis