A general sample size framework for developing or updating a clinical prediction model

2025-04-25

AI Summary

Existing sample size calculation methods for clinical prediction models inadequately address simultaneous guarantees across multiple dimensions—performance degradation, calibration, discrimination, clinical utility, prediction error, and fairness. Method: We propose the first unified Bayesian posterior simulation framework that quantifies the probability of meeting prespecified performance thresholds across these dimensions, by modeling the joint distribution of predictors in the target population and anchoring to a reference model. Contribution/Results: Our framework innovatively integrates probabilistic performance guarantees, statistical measures of calibration/discrimination degradation, instability metrics, and subgroup fairness assessment, augmented by a novel single-sample Bayesian acceleration algorithm for penalized regression. Validated via full simulation and Fisher information decomposition in preeclampsia prediction, it demonstrates that required sample sizes critically depend on key estimands and modeling strategies—substantially outperforming conventional heuristics based solely on c-statistic or event count.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

Aims: To propose a general sample size framework for developing or updating a clinical prediction model using any statistical or machine learning method, based on drawing samples from anticipated posterior distributions and targeting assurance in predictive performance. Methods: Users provide a reference model (eg, matching outcome incidence, predictor weights and c-statistic of previous models), and a (synthetic) dataset reflecting the joint distribution of candidate predictors in the target population. Then a fully simulation-based approach allows the impact of a chosen development sample size and modelling strategy to be examined. This generates thousands of models and, by applying each to the target population, leads to posterior distributions of individual predictions and model performance (degradation) metrics, to inform required sample size. To improve computation speed for penalised regression, we also propose a one-sample Bayesian analysis combining shrinkage priors with a likelihood decomposed into sample size and Fisher's information. Results: The framework is illustrated when developing pre-eclampsia prediction models using logistic regression (unpenalised, uniform shrinkage, lasso or ridge) and random forests. We show it encompasses existing sample size calculation criteria whilst providing model assurance probabilities, instability metrics and degradation statistics about calibration, discrimination, clinical utility, prediction error and fairness. Crucially, the required sample size depends on the users' key estimands and planned model development or updating approach. Conclusions: The framework generalises existing sample size proposals for model development by utilising anticipated posterior distributions conditional on a chosen sample size and development strategy. This informs the sample size required to target appropriate model performance.

Problem

• Proposing a sample size framework for clinical prediction models
• Evaluating model performance degradation with simulation-based approach
• Determining required sample size based on user-defined estimands

Innovation

• Simulation-based sample size framework for prediction models
• Combines shrinkage priors with decomposed likelihood
• Generates posterior distributions for performance metrics