LesionDiffusion: Towards Text-controlled General Lesion Synthesis

2025-03-02

AI Summary

Medical lesion detection in 3D CT relies heavily on large-scale pixel-level annotations, suffers from poor scalability, and lacks fine-grained controllability in synthesis. To address these limitations, this paper introduces the first text-controllable 3D CT lesion synthesis framework. Our approach employs a structured lesion report template-driven two-stage diffusion model (LMNet + LINet), integrating a text encoder, 3D U-Net backbone, lesion-attribute conditional guidance, and cross-modal feature alignment. The framework enables fine-grained, organ- and lesion-type-agnostic controllable synthesis and jointly generates lesion images with precise segmentation masks. Evaluated across 14 lesion categories spanning eight anatomical organs, our method significantly improves downstream segmentation performance and demonstrates zero-shot generalization capability—outperforming state-of-the-art approaches.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Fully-supervised lesion recognition methods in medical imaging face challenges due to the reliance on large annotated datasets, which are expensive and difficult to collect. To address this, synthetic lesion generation has become a promising approach. However, existing models struggle with scalability, fine-grained control over lesion attributes, and the generation of complex structures. We propose LesionDiffusion, a text-controllable lesion synthesis framework for 3D CT imaging that generates both lesions and corresponding masks. By utilizing a structured lesion report template, our model provides greater control over lesion attributes and supports a wider variety of lesion types. We introduce a dataset of 1,505 annotated CT scans with paired lesion masks and structured reports, covering 14 lesion types across 8 organs. LesionDiffusion consists of two components: a lesion mask synthesis network (LMNet) and a lesion inpainting network (LINet), both guided by lesion attributes and image features. Extensive experiments demonstrate that LesionDiffusion significantly improves segmentation performance, with strong generalization to unseen lesion types and organs, outperforming current state-of-the-art models. Code will be available at https://github.com/HengruiTianSJTU/LesionDiffusion.

Problem

• Overcomes reliance on large annotated medical datasets
• Enables fine-grained control over lesion attributes
• Generates complex lesion structures in 3D CT imaging

Innovation

• Text-controllable lesion synthesis framework
• Structured lesion report template usage
• Lesion mask and inpainting networks integration