Adjusting Initial Noise to Mitigate Memorization in Text-to-Image Diffusion Models

2025-10-08

AI Summary

Text-to-image diffusion models exhibit memorization and reproduction of training data, posing significant privacy and copyright risks. This paper identifies that the choice of initial noise critically influences the timing at which denoising trajectories escape memory-induced attractor basins. Leveraging this insight, we propose two complementary strategies—collective and individual initial noise optimization—marking the first approach to mitigate model memorization via noise initialization. Our method builds upon classifier-free guidance (CFG) and integrates empirical observations with differentiable noise optimization to dynamically adjust the initial noise, thereby promoting early departure from memorized regions. Experiments demonstrate that our approach substantially reduces training data reconstruction rates while preserving text–image alignment fidelity. Consequently, it enhances model safety in privacy- and copyright-sensitive applications.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Learning to Search
• Machine Learning: Privacy

Application Category

• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Security and Privacy: Data transparency and provenance
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Despite their impressive generative capabilities, text-to-image diffusion models often memorize and replicate training data, prompting serious concerns over privacy and copyright. Recent work has attributed this memorization to an attraction basin-a region where applying classifier-free guidance (CFG) steers the denoising trajectory toward memorized outputs-and has proposed deferring CFG application until the denoising trajectory escapes this basin. However, such delays often result in non-memorized images that are poorly aligned with the input prompts, highlighting the need to promote earlier escape so that CFG can be applied sooner in the denoising process. In this work, we show that the initial noise sample plays a crucial role in determining when this escape occurs. We empirically observe that different initial samples lead to varying escape times. Building on this insight, we propose two mitigation strategies that adjust the initial noise-either collectively or individually-to find and utilize initial samples that encourage earlier basin escape. These approaches significantly reduce memorization while preserving image-text alignment.

Problem

• Mitigate memorization in text-to-image diffusion models
• Adjust initial noise to promote earlier basin escape
• Reduce memorization while preserving image-text alignment

Innovation

• Adjust initial noise to reduce memorization
• Use collective or individual noise adjustments
• Find samples promoting earlier basin escape