A Statistical Framework for Understanding Causal Effects that Vary by Treatment Initiation Time in EHR-based Studies

2025-12-22

AI Summary

In real-world EHR studies, heterogeneous patient enrollment times and evolving treatment technologies cause observed treatment effects to vary with calendar time—yet it remains challenging to disentangle whether such variation reflects genuine changes in causal efficacy or merely shifts in patient population composition. This paper proposes an identification framework for calendar-time-specific average treatment effects (t-ATE). We introduce a novel doubly robust time-varying effect projection coupled with marginal structural model (MSM) selection, and develop a standardized-analysis-based covariate shift attribution metric to decouple population drift from true therapeutic change. Applied to a Kaiser Permanente cohort (2005–2011) of severely obese patients undergoing bariatric surgery, our method precisely quantifies the time-varying comparative effectiveness of two surgical procedures versus non-surgical management. Results indicate that approximately 40% of the observed temporal variation in treatment effects stems from baseline covariate drift—not intrinsic efficacy decay.

Technology Category

• Machine Learning: Causal Learning
• Reasoning under Uncertainty: Causality
• Knowledge Representation and Reasoning: Action, Change, and Causality

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• User Modeling, Personalization and Recommendation: Studies of user behavior, including longitudinal effects of personalized systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Comparative effectiveness studies using electronic health records (EHR) consider data from patients who could ``enter'' the study cohort at any point during an interval that spans many years in calendar time. Unlike treatments in tightly controlled trials, real-world treatments can evolve over calendar time, especially if comparators include standard of care, or procedures where techniques may improve. Efforts to assess whether treatment efficacy itself is changing are complicated by changing patient populations, with potential covariate shift in key effect modifiers. In this work, we propose a statistical framework to estimate calendar-time specific average treatment effects and describe both how and why effects vary across treatment initiation time in EHR-based studies. Our approach projects doubly robust, time-specific treatment effect estimates onto candidate marginal structural models and uses a model selection procedure to best describe how effects vary by treatment initiation time. We further introduce a novel summary metric, based on standardization analysis, to quantify the role of covariate shift in explaining observed effect changes and disentangle changes in treatment effects from changes in the patient population receiving treatment. Extensive simulations using EHR data from Kaiser Permanente are used to validate the utility of the framework, which we apply to study changes in relative weight loss following two bariatric surgical interventions versus no surgery among patients with severe obesity between 2005-2011.

Problem

• Estimates calendar-time specific average treatment effects in EHR studies
• Describes how and why treatment effects vary by initiation time
• Quantifies covariate shift role in explaining observed effect changes

Innovation

• Statistical framework estimates time-specific treatment effects
• Projects robust estimates onto marginal structural models
• Standardization metric quantifies covariate shift role