Integrated Analysis for Electronic Health Records with Structured and Sporadic Missingness

2025-06-10

AI Summary

To address the dual challenge of structured and random missingness in multi-source electronic health records (EHRs)—which degrades data utility and biases population-level health analytics—this paper proposes Macomss, the first theoretically grounded and empirically robust end-to-end joint imputation framework. Macomss uniquely integrates missingness mechanism modeling, cross-source heterogeneous data alignment, generative imputation, and downstream task-aware optimization to jointly model and impute both structured and random missing patterns. Evaluated on real-world EHR data from three hospitals within the Duke Health System, Macomss achieves significantly lower imputation error and superior downstream predictive performance compared to state-of-the-art methods. Extensive ablation studies, synthetic benchmarks, and cross-institutional validation further confirm its stability and generalizability across diverse clinical settings.

Technology Category

• Machine Learning: Multi-instance/Multi-view Learning
• Search and Optimization: Metareasoning and Metaheuristics
• Intelligent Robots: Multi-Robot Systems

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems

Abstract

Objectives: We propose a novel imputation method tailored for Electronic Health Records (EHRs) with structured and sporadic missingness. Such missingness frequently arises in the integration of heterogeneous EHR datasets for downstream clinical applications. By addressing these gaps, our method provides a practical solution for integrated analysis, enhancing data utility and advancing the understanding of population health. Materials and Methods: We begin by demonstrating structured and sporadic missing mechanisms in the integrated analysis of EHR data. Following this, we introduce a novel imputation framework, Macomss, specifically designed to handle structurally and heterogeneously occurring missing data. We establish theoretical guarantees for Macomss, ensuring its robustness in preserving the integrity and reliability of integrated analyses. To assess its empirical performance, we conduct extensive simulation studies that replicate the complex missingness patterns observed in real-world EHR systems, complemented by validation using EHR datasets from the Duke University Health System (DUHS). Results: Simulation studies show that our approach consistently outperforms existing imputation methods. Using datasets from three hospitals within DUHS, Macomss achieves the lowest imputation errors for missing data in most cases and provides superior or comparable downstream prediction performance compared to benchmark methods. Conclusions: We provide a theoretically guaranteed and practically meaningful method for imputing structured and sporadic missing data, enabling accurate and reliable integrated analysis across multiple EHR datasets. The proposed approach holds significant potential for advancing research in population health.

Problem

• Handles structured and sporadic missing data in EHRs
• Improves data utility for integrated EHR analysis
• Enhances accuracy of population health research

Innovation

• Novel imputation method for EHR missingness
• Macomss framework handles structured missing data
• Theoretical guarantees ensure robust integrated analysis