Expected value of sample information calculations for risk prediction model development

2024-10-04

AI Summary

Risk prediction models developed with limited data often deviate from the true underlying model, leading to suboptimal clinical decision-making and quantifiable utility loss—yet conventional sample size planning prioritizes statistical precision over clinical utility. Method: Grounded in decision theory, this paper introduces the Expected Value of Sample Information (EVSI) into risk prediction model development for the first time, using Net Benefit (NB) as the clinical utility metric to quantify both the utility loss attributable to finite-sample estimation and the expected utility gain from acquiring additional data. We propose a computationally feasible EVSI estimator based on bootstrap resampling integrated with decision curve analysis. Contribution/Results: Empirical case studies demonstrate that the framework enables clinically meaningful sample size decisions, shifting sample size planning from a traditional inference-accuracy paradigm to a decision-utility–driven paradigm.

Technology Category

• Reasoning under Uncertainty: Decision/Utility Theory
• Machine Learning: Calibration & Uncertainty Quantification
• Planning, Routing, and Scheduling: Model-Based Reasoning

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Models for Web evolution

Abstract

Risk prediction models are often advertised as deterministic functions that map covariates to predicted risks. However, they are typically trained using finite samples, and as such, their predictions are inherently uncertain. This uncertainty has been addressed in terms of uncertainty around metrics of model performance (e.g., confidence intervals around c-statistic), as well as uncertainty or instability of predictions. Correspondingly, sample size calculations for model development studies target the precision of estimates of summary statistics and the stability of predictions. However, when evaluating the clinical utility of a model (as in Net Benefit (NB) calculations in decision curve analysis), statistical inference is less relevant. From a decision-theoretic perspective, the finite size of the sample results in utility loss due to the discrepancy between the fitted model and the correct model. From this perspective, procuring more development data is associated with an expected gain in the utility of using the model. In this work, we define the Expected Value of Sample Information (EVSI) as the expected gain in clinical utility, defined in NB terms, by procuring an additional development sample of a given size. We propose a bootstrap-based algorithm for EVSI computations and demonstrate its feasibility and face validity in a case study. We conclude that decision-theoretic metrics can complement classical inferential methods when designing studies aimed at developing risk prediction models.

Problem

• Estimating utility loss from finite sample risk prediction models
• Calculating Expected Value of Sample Information for clinical utility
• Proposing bootstrap method to evaluate model development data needs

Innovation

• Bootstrap-based algorithm for EVSI computations
• Expected Value of Sample Information (EVSI) definition
• Decision-theoretic metrics complement classical methods