Improving Wald's (approximate) sequential probability ratio test by avoiding overshoot

2024-10-21

AI Summary

Wald’s sequential probability ratio test (SPRT) suffers from overshoot at stopping times, preventing approximate thresholds—such as ((1-eta)/alpha) and (eta/(1-alpha))—from strictly controlling Type I/II error rates (when (eta > 0)) or guaranteeing optimality (when (eta = 0)). This paper introduces “sequential boosting”, a novel method that eliminates overshoot by constructing a corrected likelihood ratio statistic. It achieves, for the first time: (1) exact (alpha)/(eta) error control with strictly smaller expected sample size than approximate SPRT when (eta > 0); (2) optimal power-one performance—matching the theoretical lower bound on expected sample size—when (eta = 0); and (3) natural generalizations to confidence sequences, sampling-without-replacement settings, and conformal martingale frameworks. Theoretical analysis proves precise error calibration, while simulations demonstrate substantial sample-size reduction. The method is plug-and-play and broadly applicable across sequential inference paradigms.

Technology Category

• Reasoning under Uncertainty: Sequential Decision Making
• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Privacy-enhancing technologies
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Wald's sequential probability ratio test (SPRT) is a cornerstone of sequential analysis. Based on desired type-I, II error levels $alpha, eta$, it stops when the likelihood ratio crosses certain thresholds, guaranteeing optimality of the expected sample size. However, these thresholds are not closed form and the test is often applied with approximate thresholds $(1-eta)/alpha$ and $eta/(1-alpha)$ (approximate SPRT). When $eta>0$, this neither guarantees error control at $alpha,eta$ nor optimality. When $eta=0$ (power-one SPRT), this method is conservative and not optimal. The looseness in both cases is caused by overshoot: the test statistic overshoots the thresholds at the stopping time. Numerically calculating thresholds may be infeasible, and most software packages do not do this. We improve the approximate SPRT by modifying the test statistic to avoid overshoot. Our `sequential boosting' technique uniformly improves power-one SPRTs $(eta=0)$ for simple nulls and alternatives, or for one-sided nulls and alternatives in exponential families. When $eta>0$, our techniques provide guaranteed error control at $alpha,eta$, while needing less samples than the approximate SPRT in our simulations. We also provide several nontrivial extensions: confidence sequences, sampling without replacement and conformal martingales.

Problem

• Avoid overshoot in Wald's approximate SPRT to ensure error control
• Improve power-one SPRT for simple or one-sided hypotheses
• Extend method to confidence sequences and non-replacement sampling

Innovation

• Modifies test statistic to avoid overshoot
• Improves power-one SPRTs for simple hypotheses
• Ensures error control with fewer samples