Testing for Underpowered Literatures

2024-06-19

AI Summary

This paper assesses the extent to which low statistical power—due to insufficient sample sizes—plagues empirical literature: how many additional statistically significant results would emerge if sample sizes were doubled? To address this, the authors develop a novel distribution-free deconvolution estimator that corrects for publication bias without imposing parametric assumptions on the true treatment effect distribution, and enables inference via asymptotic normality. Their method quantifies the “scale-up effect” through counterfactual power analysis. Applied to economics randomized controlled trials (RCTs), the analysis reveals that doubling sample sizes increases average t-test power by only 7.2 percentage points, indicating low sensitivity of current studies to scale-up. The findings support funding more small-scale experiments rather than concentrating resources on a few large ones.

Technology Category

• Game Theory and Economic Paradigms: Mechanism Design
• Reasoning under Uncertainty: Causality
• Search and Optimization: Non-convex Optimization

Application Category

• Security and Privacy: Large-scale security measurements
• Economics, Online Markets and Human Computation: Sustainability of Web economics
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

How many experimental studies would have come to different conclusions had they been run on larger samples? I show how to estimate the expected number of statistically significant results that a set of experiments would have reported had their sample sizes all been counterfactually increased. The proposed deconvolution estimator is asymptotically normal and adjusts for publication bias. Unlike related methods, this approach requires no assumptions of any kind about the distribution of true intervention treatment effects. An application to randomized trials (RCTs) published in economics journals finds that doubling every sample would increase the power of t-tests by 7.2 percentage points on average. This effect is smaller than for non-RCTs and comparable to systematic replications in laboratory psychology where previous studies enabled more accurate power calculations. This suggests that RCTs are on average relatively insensitive to sample size increases. Funders should generally consider sponsoring more experiments rather than fewer, larger ones.

Problem

• Estimating impact of larger samples on study conclusions
• Assessing statistical power without effect distribution assumptions
• Comparing RCT sensitivity to sample size vs non-RCTs

Innovation

• Deconvolution estimator for statistical significance estimation
• Asymptotically normal and adjusts for publication bias
• No assumptions on true intervention effects distribution