Language-Guided Trajectory Traversal in Disentangled Stable Diffusion Latent Space for Factorized Medical Image Generation

2025-03-30

AI Summary

To address insufficient fine-grained controllability in medical image generation, this paper proposes an unsupervised latent-space disentanglement and language-guided manipulation framework. Leveraging fine-tuned Stable Diffusion and CLIP text encoders, it pioneers the adaptation of pre-trained vision-language foundation models to medical imaging. Key anatomical structures and pathological features are automatically disentangled via principal component analysis (PCA) and gradient-based projection in the latent space. Subsequently, a language-conditioned latent trajectory optimization and interpolation scheme is introduced to enable semantically aligned, precise editing. Evaluated on chest X-ray and skin lesion datasets, the method achieves independent control over lesion location, morphology, and anatomical structure. Quantitatively, it reduces Fréchet Inception Distance (FID) by 18.7%; qualitatively, clinician assessments rate controllability at 4.6/5.0.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Text-to-image diffusion models have demonstrated a remarkable ability to generate photorealistic images from natural language prompts. These high-resolution, language-guided synthesized images are essential for the explainability of disease or exploring causal relationships. However, their potential for disentangling and controlling latent factors of variation in specialized domains like medical imaging remains under-explored. In this work, we present the first investigation of the power of pre-trained vision-language foundation models, once fine-tuned on medical image datasets, to perform latent disentanglement for factorized medical image generation and interpolation. Through extensive experiments on chest X-ray and skin datasets, we illustrate that fine-tuned, language-guided Stable Diffusion inherently learns to factorize key attributes for image generation, such as the patient's anatomical structures or disease diagnostic features. We devise a framework to identify, isolate, and manipulate key attributes through latent space trajectory traversal of generative models, facilitating precise control over medical image synthesis.

Problem

• Disentangling latent factors in medical image generation
• Controlling key attributes via language-guided diffusion models
• Enhancing precision in medical image synthesis and interpolation

Innovation

• Fine-tuned Stable Diffusion for medical images
• Latent space trajectory traversal control
• Language-guided attribute factorization