Anytime-Valid Inference Under Outcome Delay: A Design-Based Approach

2026-03-26

AI Summary

This study addresses the challenge in online experiments where treatment not only alters outcome values but also shifts their timing, with these effects potentially acting in opposing directions—rendering conventional inference methods ineffective over time. From a design-based perspective, under a framework where randomness arises solely from treatment assignment, the paper proposes a method for valid inference at any time point, targeting the cumulative sample reward as the causal estimand, and accommodating non-stationary and staggered enrollment settings. The key innovation lies in characterizing, for the first time, an augmented estimator that preserves the martingale structure under single-arm event-time filtration, thereby exposing the fundamental obstacle—namely, that treatment effect estimation errors do not form a martingale. Leveraging a union bound to construct confidence intervals, the approach achieves variance reduction without modeling unobserved future outcomes and substantially outperforms standard worst-case variance bounds when treatment induces asymmetric outcome arrival rates.

Abstract

Delayed outcomes are ubiquitous in online experimentation. When such a temporal dimension is present, treatment influences not only the outcome value but also the outcome timing, which can move in opposite directions. Motivated by the desire to continuously monitor the performance of treatment arms, we develop an anytime-valid approach to inference in the delayed outcome setting. To accommodate nonstationarity and staggered entry, we adopt a design-based framework where both the outcome timing and value are fixed potential outcomes, and randomness is introduced by treatment assignment only. We target the sample cumulative reward as a function of time, a causal estimand that avoids modeling the unobserved future, which would require strong assumptions violated by the nonstationarity and heterogeneity of our setting. We characterize exactly which augmented estimators admit a martingale structure under a specific single-arm filtration: the augmentation must activate at the event time and remain constant thereafter. This permits variance reduction via covariate adjustment while preserving the martingale property required for anytime-valid inference. We prove a fundamental negative result for the treatment effect: the estimation error is not a martingale under any filtration, arising from cross-arm covariance induced by randomized assignment. We resolve this using a union bound, showing it yields tighter intervals than the standard variance upper bound when treatment induces asymmetry in outcome arrival rates.

Problem

• delayed outcomes
• anytime-valid inference
• nonstationarity
• treatment effect
• online experimentation

Innovation

• anytime-valid inference
• delayed outcomes
• design-based inference
• martingale structure
• covariate adjustment