🤖 AI Summary
Diffusion models (DMs) pose severe privacy risks, as training images can be reconstructed from model parameters. This paper introduces the first differentially private (DP) adaptation of latent diffusion models (LDMs), proposing a latent-space localized DP fine-tuning method: DP-SGD is applied exclusively to attention modules while optimizing within the pre-trained autoencoder’s latent space, reducing trainable parameters by 90%. Our approach yields the first text-to-image generative model with rigorous end-to-end DP guarantees (ε = 2–8) at 256×256 resolution. Experiments on CIFAR-10 and LAION-400M demonstrate substantially improved privacy–utility trade-offs over full-model DP fine-tuning baselines, achieving higher image fidelity. The implementation is fully open-sourced to ensure reproducibility.
📝 Abstract
Diffusion models (DMs) are one of the most widely used generative models for producing high quality images. However, a flurry of recent papers points out that DMs are least private forms of image generators, by extracting a significant number of near-identical replicas of training images from DMs. Existing privacy-enhancing techniques for DMs, unfortunately, do not provide a good privacy-utility tradeoff. In this paper, we aim to improve the current state of DMs with differential privacy (DP) by adopting the $ extit{Latent}$ Diffusion Models (LDMs). LDMs are equipped with powerful pre-trained autoencoders that map the high-dimensional pixels into lower-dimensional latent representations, in which DMs are trained, yielding a more efficient and fast training of DMs. Rather than fine-tuning the entire LDMs, we fine-tune only the $ extit{attention}$ modules of LDMs with DP-SGD, reducing the number of trainable parameters by roughly $90%$ and achieving a better privacy-accuracy trade-off. Our approach allows us to generate realistic, high-dimensional images (256x256) conditioned on text prompts with DP guarantees, which, to the best of our knowledge, has not been attempted before. Our approach provides a promising direction for training more powerful, yet training-efficient differentially private DMs, producing high-quality DP images. Our code is available at https://anonymous.4open.science/r/DP-LDM-4525.
