A Sharp and Robust Test for Selective Reporting

2025-06-24

AI Summary

This paper addresses the robust detection of selective reporting under non-exact normality of t-statistics. We propose a consistency test based on the distance between t-curves. The core methodological innovation is a projection-type test statistic: it quantifies deviation by measuring the distance between a smoothed empirical t-curve and the theoretical t-curve family under no selective reporting, with Edgeworth expansion employed to correct for non-normality—ensuring both interpretability and robustness in large samples. The approach integrates t-curve modeling, smoothed empirical distribution estimation, and projection residual analysis. Empirical application to Brodeur et al.’s (2020) meta-dataset reveals that while t-curves for RCTs, IV, and DID estimators exhibit mild distortions, these deviations are largely attributable to approximate normality of t-statistics rather than selective reporting. Consequently, prior evidence of selective reporting in these designs is found to be fragile.

Technology Category

• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Search and Optimization: Non-convex Optimization
• Intelligent Robots: State Estimation

Application Category

• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

This paper proposes a test that is consistent against every detectable form of selective reporting and remains interpretable even when the t-scores are not exactly normal. The test statistic is the distance between the smoothed empirical t-curve and the set of all t-curves that would be possible in the absence of any selective reporting. This novel projection test can only be evaded in large meta-samples by selective reporting that also evades all other valid tests of restrictions on the t-curve. A second benefit of the projection test is that under the null we can interpret the projection residual as noise plus bias incurred from approximating the t-score's exact distribution with the normal. Applying the test to the Brodeur et al. (2020) meta-data, we find that the t-curves for RCTs, IVs, and DIDs are more distorted than could arise by chance. But an Edgeworth Expansion reveals that these distortions are small enough to be plausibly explained by the only approximate normality of the individual t-scores. The detection of selective reporting in this meta-sample is therefore more fragile than previously known.

Problem

• Detect selective reporting in research studies
• Test robustness with non-normal t-scores
• Assess t-curve distortions in meta-analysis

Innovation

• Consistent test for all selective reporting forms
• Distance-based novel projection test statistic
• Interpretable projection residual under null hypothesis