Quantifying the Internal Validity of Weighted Estimands

2024-04-22

Social Science Research Network

AI Summary

This paper addresses the internal validity quantification of weighted estimators—such as OLS, 2SLS, and TWFE—by asking: under what conditions can such estimators be rigorously interpreted as the average treatment effect (ATE) for some latent subpopulation, and what are the absolute or relative upper bounds on the size of that subpopulation? To answer this, the paper introduces the first systematic “internal validity quantification” framework, grounded in the potential outcomes model and linear projection theory. It derives necessary and sufficient conditions for the existence of a well-defined target subpopulation and provides computable upper bounds on its measure—including closed-form expressions and a general-purpose algorithm. Empirical applications reveal that conventional estimators often represent the ATE for less than 10% of the sample, thereby substantially enhancing transparency, interpretability, and reproducibility in causal inference.

Technology Category

• Machine Learning: Causal Learning
• Reasoning under Uncertainty: Causality
• Intelligent Robots: State Estimation

Application Category

• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Security and Privacy: Large-scale security measurements

Abstract

In this paper we study a class of weighted estimands, which we define as parameters that can be expressed as weighted averages of the underlying heterogeneous treatment effects. The popular ordinary least squares (OLS), two-stage least squares (2SLS), and two-way fixed effects (TWFE) estimands are all special cases within our framework. Our focus is on answering two questions concerning weighted estimands. First, under what conditions can they be interpreted as the average treatment effect for some (possibly latent) subpopulation? Second, when these conditions are satisfied, what is the upper bound on the size of that subpopulation, either in absolute terms or relative to a target population of interest? We argue that this upper bound provides a valuable diagnostic for empirical research. When a given weighted estimand corresponds to the average treatment effect for a small subset of the population of interest, we say its internal validity is low. Our paper develops practical tools to quantify the internal validity of weighted estimands.

Problem

• Conditions for weighted estimands as average treatment effects
• Upper bound on subpopulation size for treatment effects
• Tools to quantify internal validity of weighted estimands

Innovation

• Weighted estimands as heterogeneous treatment effects averages
• Conditions for average treatment effect interpretation
• Upper bound on subpopulation size as validity diagnostic