Beyond Bonferroni: Hierarchical Multiple Testing in Empirical Research

2025-07-25

AI Summary

In empirical social science and medical research, multiple hypothesis testing inflates the risk of false positives; conventional methods (e.g., Bonferroni) control the family-wise error rate (FWER) but are overly conservative, severely reducing statistical power. While sequential procedures (e.g., Holm, Hochberg) improve power, they ignore prevalent logical or causal hierarchical structures among hypotheses. This paper proposes a novel hierarchical multiple testing framework that innovatively integrates structured strategies—including fixed-sequence, fallback, and gatekeeping procedures—and systematically compares their statistical properties with classical approaches. Results demonstrate that the framework maintains strict FWER control while substantially enhancing statistical power, improving inferential interpretability and practical applicability. It thus offers a theoretically rigorous yet operationally feasible paradigm for hierarchical empirical studies, such as clinical trials and policy evaluations.

Technology Category

• Reasoning under Uncertainty: Sequential Decision Making
• Machine Learning: Statistical Relational/Logic Learning
• Game Theory and Economic Paradigms: Mechanism Design

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Large-scale security measurements

Abstract

Empirical research in the social and medical sciences frequently involves testing multiple hypotheses simultaneously, increasing the risk of false positives due to chance. Classical multiple testing procedures, such as the Bonferroni correction, control the family-wise error rate (FWER) but tend to be overly conservative, reducing statistical power. Stepwise alternatives like the Holm and Hochberg procedures offer improved power while maintaining error control under certain dependence structures. However, these standard approaches typically ignore hierarchical relationships among hypotheses - structures that are common in settings such as clinical trials and program evaluations, where outcomes are often logically or causally linked. Hierarchical multiple testing procedures - including fixed sequence, fallback, and gatekeeping methods -explicitly incorporate these relationships, providing more powerful and interpretable frameworks for inference. This paper reviews key hierarchical methods, compares their statistical properties and practical trade-offs, and discusses implications for applied empirical research.

Problem

• Addresses false positives in multiple hypothesis testing
• Improves power over conservative Bonferroni correction
• Incorporates hierarchical relationships among hypotheses

Innovation

• Hierarchical multiple testing procedures
• Fixed sequence and fallback methods
• Gatekeeping for structured hypotheses