A Multimodal Deep Learning Framework for Predicting ICU Deterioration: Integrating ECG Waveforms with Clinical Data and Clinician Benchmarking

2026-01-10

AI Summary

Current ICU risk prediction models often rely on single data sources or isolated outcomes, limiting their ability to integrate multimodal clinical information. This work proposes MDS ICU, a unified multitask deep learning framework that, for the first time, combines structured state-space models (S4) with RealMLP to effectively fuse ECG waveforms and structured clinical data—including vital signs and laboratory measurements—for continuous prediction of 33 clinically relevant outcomes using the MIMIC-IV database. The model achieves AUROC scores of 0.90–0.97 on critical endpoints such as 24-hour mortality and invasive ventilation, demonstrates excellent calibration, and significantly outperforms both clinician judgment and large language models, offering high-precision support for critical care decision-making.

Technology Category

• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: ML for personalized search and recommendations
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Artificial intelligence holds strong potential to support clinical decision making in intensive care units where timely and accurate risk assessment is critical. However, many existing models focus on isolated outcomes or limited data types, while clinicians integrate longitudinal history, real time physiology, and heterogeneous clinical information. To address this gap, we developed MDS ICU, a unified multimodal machine learning framework that fuses routinely collected data including demographics, biometrics, vital signs, laboratory values, ECG waveforms, surgical procedures, and medical device usage to provide continuous predictive support during ICU stays. Using 63001 samples from 27062 patients in MIMIC IV, we trained a deep learning architecture that combines structured state space S4 encoders for ECG waveforms with multilayer perceptron RealMLP encoders for tabular data to jointly predict 33 clinically relevant outcomes spanning mortality, organ dysfunction, medication needs, and acute deterioration. The model achieved strong discrimination with AUROCs of 0.90 for 24 hour mortality, 0.92 for sedative administration, 0.97 for invasive mechanical ventilation, and 0.93 for coagulation dysfunction. Calibration analysis showed close agreement between predicted and observed risks, with consistent gains from ECG waveform integration. Comparisons with clinicians and large language models showed that model predictions alone outperformed both, and that providing model outputs as decision support further improved their performance. These results demonstrate that multimodal AI can deliver clinically meaningful risk stratification across diverse ICU outcomes while augmenting rather than replacing clinical expertise, establishing a scalable foundation for precision critical care decision support.

Problem

• ICU deterioration
• multimodal data
• clinical decision support
• risk prediction
• ECG waveforms

Innovation

• multimodal deep learning
• ECG waveform integration
• S4 encoder
• RealMLP
• ICU deterioration prediction