🤖 AI Summary
This study addresses the challenge of accurately identifying subregions with genuine treatment effects in multi-site randomized policy experiments, where conventional multiple testing procedures often lack power. The authors propose a top-down, tree-structured sequential testing procedure that begins by evaluating the overall effect and then recursively tests groups of sites and individual sites, halting further testing within any branch once non-significance is encountered. This approach innovatively integrates a hierarchical tree framework, Hommel-type correction, and adaptive α allocation to enhance detection power for heterogeneous effects while rigorously controlling the weak familywise error rate (FWER). In simulations based on a 44-site education experiment, the method achieved a 44% detection rate—approximately four times higher than the standard Hommel procedure (11%)—and was successfully applied across 25 MDRC education trials.
📝 Abstract
Experimental evaluations of public policies often randomize a new intervention within many sites or blocks. After a report of an overall result -- statistically significant or not -- the natural question from a policy maker is: \emph{where} did any effects occur? Standard adjustments for multiple testing provide little power to answer this question. In simulations modeled after a 44-block education trial, the Hommel adjustment -- among the most powerful procedures controlling the family-wise error rate (FWER) -- detects effects in only 11\% of truly non-null blocks. We develop a procedure that tests hypotheses top-down through a tree: test the overall null at the root, then groups of blocks, then individual blocks, stopping any branch where the null is not rejected. In the same 44-block design, this approach detects effects in 44\% of non-null blocks -- roughly four times the detection rate. A stopping rule and valid tests at each node suffice for weak FWER control. We show that the strong-sense FWER depends on how rejection probabilities accumulate along paths through the tree. This yields a diagnostic: when power decays fast enough relative to branching, no adjustment is needed; otherwise, an adaptive $\alpha$-adjustment restores control. We apply the method to 25 MDRC education trials and provide an R package, \texttt{manytestsr}.