🤖 AI Summary
This paper exposes a fundamental gap between statistical significance (e.g., one-sided *p* = 0.025) and actual replicability: under identical sample sizes, the probability of replicating an effect in the same direction is only ~0.975, while the probability of reproducing statistical significance is markedly lower (~0.283). Conventional power analysis overestimates replicability by ignoring sampling variance in the original effect estimate. To address this, we develop a replication probability model grounded in variance propagation—formally integrating the sampling variances of both original and replication effect estimates. Our framework unifies frequentist and Bayesian perspectives, discarding noninformative priors in favor of discretized probability mass analysis. The resulting theory yields novel, high-confidence replication sample-size criteria, providing both theoretical foundations and practical tools for robust biomedical validation studies.
📝 Abstract
Clinicians and scientists have traditionally focussed on whether their findings will be replicated and are very familiar with the concept. The probability that a replication study yields an effect with the same sign, or the same statistical significance as an original study depends on the sum of the variances of the effect estimates. On this basis, when P equals 0.025 one-sided and the replication study has the same sample size and variance as the original study, the probability of achieving a one-sided P is less than or equal to 0.025 a second time is only about 0.283, consistent with currently observed modest replication rates. A higher replication probability would require a larger sample size than that derived from current single variance power calculations. However, if the replication study is based on an infinitely large sample size and thus has negligible variance then the probability that its estimated mean is same sign is 1 - P = 0.975. The reasoning is made clearer by changing continuous distributions to discretised scales and probability masses, thus avoiding ambiguity and improper flat priors. This perspective is consistent with Frequentist and Bayesian interpretations and also requires further reasoning when testing scientific hypotheses and making decisions.