When Does Multimodal Learning Help in Healthcare? A Benchmark on EHR and Chest X-Ray Fusion

2026-02-27

AI Summary

This study systematically investigates the conditions under which multimodal learning enhances clinical outcomes in healthcare prediction, with a focus on scenarios involving missing modalities and fairness constraints. Leveraging standardized cohorts from MIMIC-IV and MIMIC-CXR, it evaluates multiple fusion strategies—including cross-modal learning and feature concatenation—in integrating electronic health records (EHR) with longitudinal chest X-rays (CXR). The work provides the first comprehensive answers to four fundamental questions in multimodal medical AI, revealing that while fusion improves performance under complete data, it suffers significant degradation when modalities are missing. Moreover, multimodality alone does not inherently improve fairness, and modality imbalance arising from the temporal structure of EHR cannot be adequately mitigated by architectural design alone. The project releases CareBench, an open-source, extensible benchmarking toolkit.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Machine learning holds promise for advancing clinical decision support, yet it remains unclear when multimodal learning truly helps in practice, particularly under modality missingness and fairness constraints. In this work, we conduct a systematic benchmark of multimodal fusion between Electronic Health Records (EHR) and chest X-rays (CXR) on standardized cohorts from MIMIC-IV and MIMIC-CXR, aiming to answer four fundamental questions: when multimodal fusion improves clinical prediction, how different fusion strategies compare, how robust existing methods are to missing modalities, and whether multimodal models achieve algorithmic fairness. Our study reveals several key insights. Multimodal fusion improves performance when modalities are complete, with gains concentrating in diseases that require complementary information from both EHR and CXR. While cross-modal learning mechanisms capture clinically meaningful dependencies beyond simple concatenation, the rich temporal structure of EHR introduces strong modality imbalance that architectural complexity alone cannot overcome. Under realistic missingness, multimodal benefits rapidly degrade unless models are explicitly designed to handle incomplete inputs. Moreover, multimodal fusion does not inherently improve fairness, with subgroup disparities mainly arising from unequal sensitivity across demographic groups. To support reproducible and extensible evaluation, we further release a flexible benchmarking toolkit that enables plug-and-play integration of new models and datasets. Together, this work provides actionable guidance on when multimodal learning helps, when it fails, and why, laying the foundation for developing clinically deployable multimodal systems that are both effective and reliable. The open-source toolkit can be found at https://github.com/jakeykj/CareBench.

Problem

• multimodal learning
• healthcare
• modality missingness
• algorithmic fairness
• clinical prediction

Innovation

• multimodal learning
• clinical decision support
• modality missingness
• algorithmic fairness
• benchmarking toolkit