The Bottom-Up Approach for Powerful Testing with FWER Control

2025-11-17

AI Summary

Balancing statistical power and computational efficiency remains challenging in multiple testing, particularly under stringent family-wise error rate (FWER) control. Method: We propose a bottom-up construction framework grounded in closed testing, the first to embed a global power-optimization objective directly into the design of local tests at each hierarchical level. This yields a novel testing procedure that satisfies consonance and strictly controls FWER. By recursively constructing analytically tractable and computationally feasible test statistics, our method ensures statistical rigor while markedly improving computational scalability. Contribution/Results: Extensive simulations and real-data analyses demonstrate that our procedure achieves significantly higher true positive rates than leading competitors—including Holm, Hochberg, and existing closed testing variants—while maintaining exact FWER control. Theoretical analysis confirms its optimality properties, and empirical results underscore its practical utility across diverse applications.

Technology Category

• Search and Optimization: Mixed Discrete/Continuous Search
• Reasoning under Uncertainty: Stochastic Optimization
• Machine Learning: Ensemble Methods

Application Category

• Security and Privacy: Large-scale security measurements
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

We seek to design novel multiple testing procedures, which take into account a relevant notion of ''power'' or true discovery on the one hand, and allow computationally efficient test design and application on the other. Towards this end we characterize the optimal procedures that strongly control the family-wise error rate, for a range of power objectives measuring the success of multiple testing procedures in making true individual discoveries, and under a reasonable set of assumptions. While we cannot generally find these optimal solutions in practice, we propose the bottom-up approach, which constructs consonant closed testing procedures, while taking into account the overall power objective in designing the tests on every level of the closed testing hierarchy. This leads to a general recipe, yielding novel procedures which are computationally practical and demonstrate substantially improved power in both simulations and a real data study, compared to existing procedures.

Problem

• Design novel multiple testing procedures with strong FWER control
• Optimize power objectives for true individual discoveries efficiently
• Propose bottom-up approach for computationally practical improved power

Innovation

• Bottom-up approach constructs consonant closed testing
• Optimizes power objectives across closed testing hierarchy
• Yields computationally practical procedures with improved power