An online generalization of the (e-)Benjamini-Hochberg procedure

2024-07-30

AI Summary

Classical Benjamini–Hochberg (BH) and e-BH procedures—designed for offline multiple testing—are not directly applicable to streaming data, and existing online FDR control methods lack rigorous guarantees for stopping-time FDR. Method: We propose an online BH/e-BH framework incorporating a *delayed rejection* mechanism. Contribution/Results: Our method is the first to simultaneously achieve theoretical consistency—provably controlling both fixed-time and stopping-time FDR under arbitrary dependence—and reduction—recovering standard offline BH/e-BH as special cases. We unify and extend the analysis of major online procedures, proving their stopping-time FDR control via sequential decision theory, dependence-agnostic bounding, and stopping-time extension techniques. Crucially, our framework attains the *same* FDR bound as offline BH/e-BH. An open-source implementation enables real-time streaming hypothesis testing.

Technology Category

• Machine Learning: Online Learning & Bandits
• Reasoning under Uncertainty: Sequential Decision Making
• Data Mining & Knowledge Management: Data Stream Mining

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications

Abstract

In online multiple testing, the hypotheses arrive one by one, and at each time we must immediately reject or accept the current hypothesis solely based on the data and hypotheses observed so far. Many online procedures have been proposed, but none of them are generalizations of the Benjamini-Hochberg (BH) procedure based on p-values, or of the e-BH procedure that uses e-values. In this paper, we consider a relaxed problem setup that allows the current hypothesis to be rejected at any later step. We show that this relaxation allows us to define -- what we justify extensively to be -- the natural and appropriate online extension of the BH and e-BH procedures. We show that the FDR guarantees for BH (resp. e-BH) and online BH (resp. online e-BH) are identical under positive, negative or arbitrary dependence, at fixed and stopping times. Further, the online BH (resp. online e-BH) rule recovers the BH (resp. e-BH) rule as a special case when the number of hypotheses is known to be fixed. Of independent interest, our proof techniques also allow us to prove that numerous existing online procedures, which were known to control the FDR at fixed times, also control the FDR at stopping times.

Problem

• Online extension of Benjamini-Hochberg procedure
• FDR control for sequential hypothesis testing
• Generalizing BH and e-BH methods online

Innovation

• Online extension of BH and e-BH procedures
• FDR guarantees identical under various dependencies
• Extends Closure Principle to online FDR control