CXR-TFT: Multi-Modal Temporal Fusion Transformer for Predicting Chest X-ray Trajectories

2025-07-19

AI Summary

Intensive care unit (ICU) patients undergo infrequent chest X-ray (CXR) examinations, resulting in sparse imaging data, while existing models lack explicit temporal modeling capabilities for longitudinal CXR findings. Method: We propose the first multimodal temporal fusion framework for predicting CXR finding trajectories—jointly leveraging sparse CXR images, structured radiology reports, and high-frequency clinical time series (vital signs, laboratory values, respiratory parameters). A vision encoder extracts image representations; latent-space alignment and linear interpolation enable cross-modal and cross-sampling-rate temporal synchronization; and a conditional Transformer captures dynamic interdependencies for hour-level continuous prediction. Results: Evaluated on a retrospective cohort of 20,000 ICU patients, our model accurately forecasts abnormal CXR findings—including pulmonary edema and infiltrates—up to 12 hours before radiographic manifestation, significantly preceding visible imaging changes and providing a critical intervention window for acute respiratory distress syndrome (ARDS) and other life-threatening conditions.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

In intensive care units (ICUs), patients with complex clinical conditions require vigilant monitoring and prompt interventions. Chest X-rays (CXRs) are a vital diagnostic tool, providing insights into clinical trajectories, but their irregular acquisition limits their utility. Existing tools for CXR interpretation are constrained by cross-sectional analysis, failing to capture temporal dynamics. To address this, we introduce CXR-TFT, a novel multi-modal framework that integrates temporally sparse CXR imaging and radiology reports with high-frequency clinical data, such as vital signs, laboratory values, and respiratory flow sheets, to predict the trajectory of CXR findings in critically ill patients. CXR-TFT leverages latent embeddings from a vision encoder that are temporally aligned with hourly clinical data through interpolation. A transformer model is then trained to predict CXR embeddings at each hour, conditioned on previous embeddings and clinical measurements. In a retrospective study of 20,000 ICU patients, CXR-TFT demonstrated high accuracy in forecasting abnormal CXR findings up to 12 hours before they became radiographically evident. This predictive capability in clinical data holds significant potential for enhancing the management of time-sensitive conditions like acute respiratory distress syndrome, where early intervention is crucial and diagnoses are often delayed. By providing distinctive temporal resolution in prognostic CXR analysis, CXR-TFT offers actionable 'whole patient' insights that can directly improve clinical outcomes.

Problem

• Predicts CXR trajectories in ICU patients using multi-modal data
• Overcomes irregular CXR acquisition and cross-sectional analysis limits
• Enables early detection of abnormal CXR findings for timely intervention

Innovation

• Multi-modal fusion of CXR and clinical data
• Transformer model for temporal CXR prediction
• Latent embeddings aligned with hourly data