Bayesian sample size calculations for external validation studies of risk prediction models

2025-04-22

AI Summary

Conventional sample size calculations for external validation of risk prediction models rely on fixed prior assumptions about model performance (e.g., calibration, discrimination) and net benefit (NB), failing to capture real-world uncertainty; moreover, traditional precision-oriented approaches—based on confidence interval width—bear weak relevance to clinical utility (NB). Method: This paper introduces, for the first time, a systematic Bayesian framework for external validation sample size determination. It constructs a joint risk–outcome distribution from prior performance summaries and proposes a multi-objective Bayesian sample size rule that jointly optimizes expected precision, assurance probability, optimal NB identification, and expected value of sample information (EVSI). Contribution/Results: The method enables decision-making under quantified uncertainty, enhancing statistical robustness and clinical relevance. Applied to external validation of a COVID-19 deterioration risk model, it improves resource efficiency and reliability of validation conclusions.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Summary: Contemporary sample size calculations for external validation of risk prediction models require users to specify fixed values of assumed model performance metrics alongside target precision levels (e.g., 95% CI widths). However, due to the finite samples of previous studies, our knowledge of true model performance in the target population is uncertain, and so choosing fixed values represents an incomplete picture. As well, for net benefit (NB) as a measure of clinical utility, the relevance of conventional precision-based inference is doubtful. In this work, we propose a general Bayesian algorithm for constructing the joint distribution of predicted risks and response values based on summary statistics of model performance in previous studies. For statistical metrics of performance, we propose sample size determination rules that either target desired expected precision, or a desired assurance probability that the precision criteria will be satisfied. For NB, we propose rules based on optimality assurance (the probability that the planned study correctly identifies the most beneficial strategy) and the Expected Value of Sample Information (EVSI), the expected gain in NB from the planned validation study. We showcase these developments in a case study on the validation of a risk prediction model for deterioration of hospitalized COVID-19 patients. Compared to the conventional sample size calculation methods, a Bayesian approach requires explicit quantification of uncertainty around model performance, but thereby enables various sample size rules based on expected precision, assurance probabilities, and value of information.

Problem

• Bayesian sample size calculations for uncertain model performance
• Addressing limitations of conventional precision-based inference for net benefit
• Proposing rules for sample size based on assurance probabilities and EVSI

Innovation

• Bayesian algorithm for joint distribution construction
• Sample size rules based on assurance probabilities
• Expected Value of Sample Information for clinical utility