MedCoDi-M: A Multi-Prompt Foundation Model for Multimodal Medical Data Generation

2025-01-08

AI Summary

To address the challenges of multimodal fusion and weak diagnostic support in clinical AI deployment—stemming from data scarcity and privacy constraints—this work introduces the first 6.77B-parameter multimodal foundation model tailored for radiological practice. We propose a novel Multi-Prompt training paradigm that establishes a unified, contrastive learning–driven latent space, enabling synchronous cross-modal generation and semantic alignment between X-ray images and radiology reports. Technically, the model integrates Latent Diffusion Models (LDMs), large-scale medical multimodal pretraining, and conditional control mechanisms. On MIMIC-CXR, it significantly outperforms five state-of-the-art baselines. Its clinical plausibility is validated via a radiologist-conducted Visual Turing Test. Moreover, the model demonstrates robust generalization under challenging real-world conditions—including data anonymization, few-shot learning, and severe class imbalance—highlighting its practical viability for clinical deployment.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Artificial Intelligence is revolutionizing medical practice, enhancing diagnostic accuracy and healthcare delivery. However, its adaptation in medical settings still faces significant challenges, related to data availability and privacy constraints. Synthetic data has emerged as a promising solution to mitigate these issues, addressing data scarcity while preserving privacy. Recently, Latent Diffusion Models have emerged as a powerful tool for generating high-quality synthetic data. Meanwhile, the integration of different modalities has gained interest, emphasizing the need of models capable of handle multimodal medical data.Existing approaches struggle to integrate complementary information and lack the ability to generate modalities simultaneously. To address this challenge, we present MedCoDi-M, a 6.77-billion-parameter model, designed for multimodal medical data generation, that, following Foundation Model paradigm, exploits contrastive learning and large quantity of data to build a shared latent space which capture the relationships between different data modalities. Further, we introduce the Multi-Prompt training technique, which significantly boosts MedCoDi-M's generation under different settings. We extensively validate MedCoDi-M: first we benchmark it against five competitors on the MIMIC-CXR dataset, a state-of-the-art dataset for Chest X-ray and radiological report generation. Secondly, we perform a Visual Turing Test with expert radiologists to assess the realism and clinical relevance of the generated data, ensuring alignment with real-world scenarios. Finally, we assess the utility of MedCoDi-M in addressing key challenges in the medical field, such as anonymization, data scarcity and imbalance learning. The results are promising, demonstrating the applicability of MedCoDi-M in medical contexts. Project page is at https://cosbidev.github.io/MedCoDi-M/.

Problem

• Medical AI
• Data scarcity
• Privacy protection

Innovation

• Medical Data Generation
• Contrastive Learning
• Privacy-preserving Diversity