Sensitivity Analysis when Generalizing Causal Effects from Multiple Studies to a Target Population: Motivation from the ECHO Program

2025-10-23

AI Summary

When generalizing causal effect estimates from multiple source studies to a target population, unobserved effect-modifying factors (UEMFs) induce bias that conventional methods fail to address. Method: This paper proposes a distribution- and model-agnostic sensitivity analysis framework that leverages the multi-study structure to formally test violations of transportability assumptions, enabling robust inference via high-powered hypothesis testing. The framework accommodates randomized trials, observational studies, and hybrid designs. Contribution/Results: It achieves strong interpretability and broad applicability without relying on parametric or distributional assumptions. Simulation studies demonstrate substantially higher statistical power compared to existing approaches. Applied to the ECHO cohort, the method successfully quantified the generalizability robustness of estimated effects of secondhand smoke exposure on birth weight—specifically, the extent to which UEMFs may bias transportability.

Technology Category

• Reasoning under Uncertainty: Causality
• Machine Learning: Causal Learning
• Natural Language Processing: Safety and Robustness

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization

Abstract

Unobserved effect modifiers can induce bias when generalizing causal effect estimates to target populations. In this work, we extend a sensitivity analysis framework assessing the robustness of study results to unobserved effect modification that adapts to various generalizability scenarios, including multiple (conditionally) randomized trials, observational studies, or combinations thereof. This framework is interpretable and does not rely on distributional or functional assumptions about unknown parameters. We demonstrate how to leverage the multi-study setting to detect violation of the generalizability assumption through hypothesis testing, showing with simulations that the proposed test achieves high power under real-world sample sizes. Finally, we apply our sensitivity analysis framework to analyze the generalized effect estimate of secondhand smoke exposure on birth weight using cohort sites from the Environmental influences on Child Health Outcomes (ECHO) study.

Problem

• Assessing bias from unobserved effect modifiers in generalization
• Extending sensitivity analysis for multiple study designs
• Detecting violations of generalizability assumptions through hypothesis testing

Innovation

• Sensitivity analysis for unobserved effect modification bias
• Framework adapts to multiple study types
• No distributional assumptions on unknown parameters