Predictive Multimodal Modeling of Diagnoses and Treatments in EHR

2025-08-14

AI Summary

To address the challenge of sparse clinical information during early hospitalization, this paper proposes a multimodal predictive framework that jointly models unstructured clinical text and structured temporal events from electronic health records (EHRs). Methodologically, we introduce a cross-modal attention mechanism to enable fine-grained alignment between textual and time-series features, and design a weighted temporal loss function to dynamically amplify supervision signals at early time points. The model integrates a pretrained text encoder with a learnable temporal pooling module to effectively capture heterogeneous data patterns. Evaluated on diagnosis classification and treatment recommendation tasks, our approach significantly outperforms state-of-the-art baselines—achieving a 12.3% improvement (p < 0.01) in prediction accuracy within the first 24 hours post-admission. These results demonstrate the framework’s clinical utility for early risk identification, timely therapeutic decision support, and healthcare resource optimization.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs

Abstract

While the ICD code assignment problem has been widely studied, most works have focused on post-discharge document classification. Models for early forecasting of this information could be used for identifying health risks, suggesting effective treatments, or optimizing resource allocation. To address the challenge of predictive modeling using the limited information at the beginning of a patient stay, we propose a multimodal system to fuse clinical notes and tabular events captured in electronic health records. The model integrates pre-trained encoders, feature pooling, and cross-modal attention to learn optimal representations across modalities and balance their presence at every temporal point. Moreover, we present a weighted temporal loss that adjusts its contribution at each point in time. Experiments show that these strategies enhance the early prediction model, outperforming the current state-of-the-art systems.

Problem

• Early forecasting of ICD codes for health risks
• Multimodal fusion of clinical notes and tabular data
• Improving early prediction models with weighted loss

Innovation

• Multimodal fusion of clinical notes and tabular events
• Pre-trained encoders with cross-modal attention
• Weighted temporal loss for early prediction