🤖 AI Summary
This study addresses the uncontrolled false positive rates and misleading inferences arising from commonly used simulation methods in shift-share designs. We systematically evaluate prevailing inferential approaches in empirical research through a suite of multilevel simulation experiments. By comparing Monte Carlo analysis with counterfactual data-generating mechanisms, we uncover non-monotonic trade-offs among fidelity, sensitivity, and risk of misdirection across simulation designs. We propose a novel “progressive-fidelity simulation framework,” demonstrating that low-fidelity simulations suffice to expose fundamental inferential flaws, whereas high-fidelity simulations detect subtle, previously overlooked biases—substantially improving detection power. The framework balances interpretability and computational efficiency, offering a reproducible and scalable paradigm for assessing the robustness of causal inference methods.
📝 Abstract
We analyze different types of simulations that applied researchers may use to assess their inference methods. We show that different types of simulations vary in many dimensions when considered as inference assessments. Moreover, we show that natural ways of running simulations may lead to misleading conclusions, and we propose alternatives. We then provide evidence that even some simple assessments can detect problems in many different settings. Alternative assessments that potentially better approximate the true data generating process may detect problems that simpler assessments would not detect. However, they are not uniformly dominant in this dimension, and may imply some costs.