Safe, Always-Valid Alpha-Investing Rules For Doubly Sequential Online Inference

2025-12-13

AI Summary

Online multiple testing under dual sequential data and task streams—arising in dynamic domains such as marketing, finance, and drug discovery—poses challenges in controlling false selection rate (FSR) while accommodating real-time decision-making and resource constraints. Method: This paper proposes the Safe and Always-Valid Alpha-investing (SAVA) framework, the first to unify always-valid p-values, e-processes, online false discovery rate (FDR) control, and dynamic α-reinvestment. Contribution/Results: SAVA guarantees strict, non-asymptotic FSR control at all times—overcoming the “alpha-death” limitation of prior methods. We prove its FSR control holds both finitely and asymptotically. Empirically, SAVA achieves significantly higher statistical power than state-of-the-art online testing methods, while ensuring reliability of real-time decisions and efficient allocation of testing resources.

Technology Category

• Machine Learning: Online Learning & Bandits
• Reasoning under Uncertainty: Sequential Decision Making
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Economics, Online Markets and Human Computation: Sustainability of Web economics
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Large-scale security measurements

Abstract

Dynamic decision-making in rapidly evolving research domains, including marketing, finance, and pharmaceutical development, presents a significant challenge. Researchers frequently confront the need for real-time action within a doubly sequential framework characterized by the continuous influx of high-volume data streams and the intermittent arrival of novel tasks. This calls for the development and implementation of new online inference protocols capable of handling both the continuous processing of incoming information and the efficient allocation of resources to address emerging priorities. We introduce a novel class of Safe and Always-Valid Alpha-investing (SAVA) rules that leverages powerful tools including always valid p-values, e-processes, and online false discovery rate methods. The SAVA algorithm effectively integrates information across all tasks, mitigates the alpha-death problem, and controls the false selection rate (FSR) at all decision points. We validate the efficacy of the SAVA framework through rigorous theoretical analysis and extensive numerical experiments. Our results demonstrate that SAVA not only offers effective control of the FSR but also significantly improves statistical power compared to traditional online testing approaches.

Problem

• Develops safe online inference rules for dynamic decision-making
• Addresses alpha-death in sequential data and task environments
• Controls false selection rate while enhancing statistical power

Innovation

• SAVA rules use always-valid p-values and e-processes
• Integrates information across tasks to mitigate alpha-death
• Controls false selection rate at all decision points