Observing the unobserved confounding through its effects: toward randomized trial-like estimates from real-world survival data

2026-04-13

AI Summary

This study addresses the pervasive challenge of unmeasured confounding in real-world survival data, which often biases treatment effect estimation. To mitigate this issue, the authors propose a three-step causal inference framework: first, they infer the aggregate effect of a latent prognostic factor \( U \) using differences in restricted mean survival time (RMST); second, they jointly balance observed covariates and \( U \) through a combination of prognostic matching, entropy balancing, and inverse probability weighting; and third, they estimate the treatment hazard ratio using multivariable survival models. This approach uniquely integrates RMST-driven inference of unmeasured confounding into the balancing strategy, substantially improving estimation accuracy across multiple cohorts—reducing log-hazard ratio error by up to tenfold, aligning observational estimates more closely with those from randomized controlled trials, and markedly decreasing inter-center heterogeneity.

Technology Category

• Machine Learning: Causal Learning
• Reasoning under Uncertainty: Causality
• Planning, Routing, and Scheduling: Scheduling under Uncertainty

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Background: Randomized controlled trials (RCTs) are costly, time-consuming, and often infeasible, while treatment-effect estimation from observational data is limited by unobserved confounding. Methods: We developed a three-step framework to address unobserved confounding in observational survival data. First, we infer a latent prognostic factor (U) from restricted mean survival time (RMST) discrepancies between patients with similar observed factors, the same treatment, and divergent outcomes, leveraging the idea that the aggregate effect of unmeasured factors can be inferred even if individual factors cannot. Second, we balance U with observed baseline covariates using prognostic matching, entropy balancing, or inverse probability of treatment weighting. Third, we apply multivariable survival analysis to estimate hazard ratios (HRs). We evaluated the framework in three observational cohorts with RCT benchmarks, two RCT cohorts, and six multicenter observational cohorts. Results: In three observational cohorts (nine comparisons), balancing U improved agreement with trial HRs in all cases; in the strongest settings, it reduced absolute log-HR error by approximately ten-fold versus using observed covariates alone (mean reduction 0.344; p=0.001). In two RCT cohorts, U was balanced across arms (most SMDs <0.1) and adjustment had minimal impact on log-HRs (mean absolute change 0.08). Across six multicenter cohorts, balancing U within centers reduced cross-center dispersion in chemotherapy log-HR estimates (mean reduction 0.147; p=0.016); when populations were directly balanced across centers to account for case-mix differences, cross-center survival differences were narrowed in 75%-100% of comparisons. Conclusions: Inferring and balancing a latent prognostic signal may reduce unobserved confounding and improve treatment-effect estimation from real-world data.

Problem

• unobserved confounding
• treatment-effect estimation
• observational survival data
• real-world data
• causal inference

Innovation

• unobserved confounding
• latent prognostic factor
• restricted mean survival time
• real-world data
• treatment-effect estimation