Reducing Diffusion Model Memorization with Higher Order Langevin Dynamics

2026-05-18

AI Summary

Diffusion models are prone to memorizing training samples, posing privacy and copyright risks. This work proposes High-Order Langevin Dynamics (HOLD) regularization, which mitigates such memorization by constraining data trajectories through auxiliary variables representing velocity, acceleration, and higher-order derivatives. Theoretical analysis reveals, for the first time, that HOLD is equivalent to applying a low-pass filter to the score function, with smoothing strength increasing as the order of dynamics rises—thereby alleviating memorization while preventing distributional collapse. Empirical results on real-world datasets demonstrate that HOLD significantly reduces memorization tendencies, with performance consistently improving as the order of the dynamics increases.

Abstract

Diffusion/score-based models have emerged as powerful generative models, capable of generating high-quality samples that mimic the training data distribution. However, it has been observed that they are prone to reproducing training samples-known as "memorization"-potentially violating copyright and privacy. In this paper, we study the effect of Higher-Order Langevin Dynamics (HOLD) on this phenomenon. HOLD diffusion processes introduce auxiliary variables; if the data variable is interpreted as "position," then the auxiliary variables can be interpreted as "velocity" and "acceleration," depending on the chosen order of the model. They were originally proposed based on the intuition that they regularize the trajectories of the data variable by implicitly imposing additional dynamical constraints. Our work provides, to our knowledge, the first theoretical characterization of the regularization effect of HOLD. Specifically, we show that in HOLD, the dynamics of the data variable are governed by a low-pass-filtered version of the learned score function, with smoothness increasing with the order of HOLD. We then analyze the optimal empirical score and the possibility of distribution collapse. Together, our results explain the mitigation of memorization as the model order increases. Finally, we present an empirical study on real-world data that supports our theory and highlights this distinct advantage of HOLD over standard diffusion in practice.

Problem

• memorization
• diffusion models
• privacy
• copyright
• score-based models

Innovation

• Higher-Order Langevin Dynamics
• Diffusion Models
• Memorization Mitigation
• Score Function Regularization
• Low-Pass Filtering