A New Design-Based Variance Estimator for Finely Stratified Experiments

2025-03-13

AI Summary

This paper addresses the challenge of design-based inference for the average treatment effect (ATE) in finely stratified randomized experiments—particularly under the extreme stratification regime where each stratum contains only one treated or one control unit. We propose a novel pairwise-differenced-mean variance estimator that pairs adjacent, similar strata. Unlike existing estimators, ours remains well-defined and upwardly biased with controllable magnitude even in the single-unit-per-stratum limit. Under a similarity assumption on adjacent strata, we prove analytically that our estimator exhibits reduced bias and is asymptotically superior to state-of-the-art alternatives. Finite-population bias analysis and i.i.d. superpopulation modeling, corroborated by Monte Carlo simulations, demonstrate that under high-quality stratification, our method yields substantially narrower confidence intervals and improved inferential accuracy. Our key contribution is the first variance estimation framework that simultaneously ensures theoretical rigor—via finite-sample bias characterization and asymptotic dominance—and practical robustness across realistic stratification scenarios.

Technology Category

• Reasoning under Uncertainty: Probabilistic Inference
• Machine Learning: Calibration & Uncertainty Quantification
• Intelligent Robots: State Estimation

Application Category

• Security and Privacy: Large-scale security measurements
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience

Abstract

This paper considers the problem of design-based inference for the average treatment effect in finely stratified experiments. Here, by"design-based'' we mean that the only source of uncertainty stems from the randomness in treatment assignment; by"finely stratified'' we mean that units are stratified into groups of a fixed size according to baseline covariates and then, within each group, a fixed number of units are assigned uniformly at random to treatment and the remainder to control. In this setting we present a novel estimator of the variance of the difference-in-means based on pairing"adjacent"strata. Importantly, our estimator is well defined even in the challenging setting where there is exactly one treated or control unit per stratum. We prove that our estimator is upward-biased, and thus can be used for inference under mild restrictions on the finite population. We compare our estimator with some well-known estimators that have been proposed previously in this setting, and demonstrate that, while these estimators are also upward-biased, our estimator has smaller bias and therefore leads to more precise inferences whenever adjacent strata are sufficiently similar. To further understand when our estimator leads to more precise inferences, we introduce a framework motivated by a thought experiment in which the finite population is modeled as having been drawn once in an i.i.d. fashion from a well-behaved probability distribution. In this framework, we argue that our estimator dominates the others in terms of limiting bias and that these improvements are strict except under strong restrictions on the treatment effects. Finally, we illustrate the practical relevance of our theoretical results through a simulation study, which reveals that our estimator can in fact lead to substantially more precise inferences, especially when the quality of stratification is high.

Problem

• Estimating variance in finely stratified experiments
• Improving inference precision with novel estimator
• Comparing bias with existing variance estimators

Innovation

• Novel variance estimator using adjacent strata pairing
• Upward-biased estimator for precise treatment effect inference
• Improved bias and precision in finely stratified experiments