Minimum Specification Perturbation: Robustness as Distance-to-Falsification in Causal Inference

2026-05-02

AI Summary

This work proposes Minimal Specification Perturbation (MSP), a novel metric for quantifying the robustness of causal inference, defined as the minimum number of analytical decisions that must be altered to render a causal effect’s confidence interval include zero—thereby formalizing robustness as a discrete distance to falsification. MSP constitutes the first distance-based robustness measure framed within a discrete perturbation paradigm, uncovering vulnerability dimensions orthogonal to existing approaches based on continuous perturbations or the Fragility Index. Leveraging combinatorial optimization, permutation calibration, and exact algorithms under additive structures—and establishing its NP-hardness—MSP effectively identifies fragile findings in the LaLonde benchmark: an MSP of 1 indicates that a single decision change suffices to overturn statistical significance, substantially reducing false positive rates in settings with weak treatment effects.

Abstract

Empirical causal claims depend on many analyst decisions, from selecting covariates to choosing estimators. Existing robustness tools summarize how results vary across these choices, but, to the best of our knowledge, do not answer: \textbf{How many analyst decisions must change to reach a specification, which is a set of choices, whose confidence interval (CI) contains zero?} We introduce \emph{Minimum Specification Perturbation (MSP)}, the smallest number of changes. MSP is small under the null, grows with effect strength and captures distance-to-falsification information that dispersion-based summaries cannot report; when making decisions under weak effects, an MSP-based rule yields lower false-positive rates than dispersion-based rules. We show that Fragility Index and MSP measure orthogonal vulnerabilities: fragility to influential observations need not imply fragility to specification choices. On the LaLonde benchmark, MSP = 1 implies that one decision change makes the CI contain zero. We further provide exact permutation calibration under randomization and characterize computation, showing tractable cases under additive structure and NP-hardness in general.

Problem

• causal inference
• robustness
• specification perturbation
• distance-to-falsification
• analyst decisions

Innovation

• Minimum Specification Perturbation
• causal inference
• robustness
• distance-to-falsification
• specification search