Predicting fixed-sample test decisions enables anytime-valid inference

2026-02-14

AI Summary

Traditional hypothesis testing relies on fixed sample sizes and struggles to accommodate sequentially arriving data, while existing sequential methods either require rigid pre-specified analysis plans or compromise statistical power. This work proposes a general framework that transforms any fixed-sample test into a sequentially valid test usable at arbitrary stopping times. By modeling future observations as missing data and predicting, under the null hypothesis, the probability that the full-sample test would reject, the method constructs an adaptive stopping rule. It requires no pre-specified analysis plan, rigorously controls the Type I error rate, and achieves near-optimal statistical power. Moreover, under the alternative hypothesis, it substantially reduces the required sample size, making it especially suitable for applications such as clinical trials where early stopping for efficacy or futility is critical.

Technology Category

• Reasoning under Uncertainty: Sequential Decision Making
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Active Learning

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Security and Privacy: Data transparency and provenance

Abstract

Statistical hypothesis tests typically use prespecified sample sizes, yet data often arrive sequentially. Interim analyses invalidate classical error guarantees, while existing sequential methods require rigid testing preschedules or incur substantial losses in statistical power. We introduce a simple procedure that transforms any fixed-sample hypothesis test into an anytime-valid test while ensuring Type-I error control and near-optimal power with substantial sample savings when the null hypothesis is false. At each step, the procedure predicts the probability that a classical test would reject the null hypothesis at its fixed-sample size, treating future observations as missing data under the null hypothesis. Thresholding this probability yields an anytime-valid stopping rule. In areas such as clinical trials, stopping early and safely can ensure that subjects receive the best treatments and accelerate the development of effective therapies.

Problem

• anytime-valid inference
• sequential testing
• Type-I error control
• statistical power
• fixed-sample test

Innovation

• anytime-valid inference
• sequential hypothesis testing
• Type-I error control
• sample efficiency
• missing data imputation