seg2med: a segmentation-based medical image generation framework using denoising diffusion probabilistic models

2025-04-12

AI Summary

This work addresses the controlled generation of high-fidelity CT/MR medical images from anatomical segmentation masks. We propose the first conditional generative framework that deeply integrates fine-grained anatomical priors with denoising diffusion probabilistic models (DDPMs). Conditioning on subject-level whole-body segmentation masks generated by TotalSegmentator, our method enables single-mask-driven joint CT/MR synthesis and bidirectional cross-modal translation (CT↔MR). Its key innovation lies in explicitly embedding high-resolution anatomical structure information into the diffusion process, thereby ensuring both anatomical fidelity and modality consistency in synthesized images. Evaluated on the XCAT digital phantom: CT and MR synthesis achieve SSIM scores of 0.94±0.02 and 0.89±0.04, respectively; CT→MR and MR→CT translation yield SSIM of 0.91±0.03 and 0.77±0.04; Dice scores exceed 0.80 for 34 of 59 organs, with an average of >0.90 across 11 core abdominal organs; FID is 3.62.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

In this study, we present seg2med, an advanced medical image synthesis framework that uses Denoising Diffusion Probabilistic Models (DDPM) to generate high-quality synthetic medical images conditioned on anatomical masks from TotalSegmentator. The framework synthesizes CT and MR images from segmentation masks derived from real patient data and XCAT digital phantoms, achieving a Structural Similarity Index Measure (SSIM) of 0.94 +/- 0.02 for CT and 0.89 +/- 0.04 for MR images compared to ground-truth images of real patients. It also achieves a Feature Similarity Index Measure (FSIM) of 0.78 +/- 0.04 for CT images from XCAT. The generative quality is further supported by a Fr'echet Inception Distance (FID) of 3.62 for CT image generation. Additionally, seg2med can generate paired CT and MR images with consistent anatomical structures and convert images between CT and MR modalities, achieving SSIM values of 0.91 +/- 0.03 for MR-to-CT and 0.77 +/- 0.04 for CT-to-MR conversion. Despite the limitations of incomplete anatomical details in segmentation masks, the framework shows strong performance in cross-modality synthesis and multimodal imaging. seg2med also demonstrates high anatomical fidelity in CT synthesis, achieving a mean Dice coefficient greater than 0.90 for 11 abdominal organs and greater than 0.80 for 34 organs out of 59 in 58 test cases. The highest Dice of 0.96 +/- 0.01 was recorded for the right scapula. Leveraging the TotalSegmentator toolkit, seg2med enables segmentation mask generation across diverse datasets, supporting applications in clinical imaging, data augmentation, multimodal synthesis, and diagnostic algorithm development.

Problem

• Generates synthetic CT and MR images from segmentation masks
• Converts medical images between CT and MR modalities
• Ensures anatomical consistency in multimodal image synthesis

Innovation

• Uses DDPM for medical image synthesis
• Generates CT and MR from segmentation masks
• Achieves high SSIM and FID scores