Memorization and Regularization in Generative Diffusion Models

2025-01-27

AI Summary

This work identifies that diffusion models, driven by empirical loss minimization in score matching, degenerate into time-varying Gaussian mixtures—leading to excessive memorization of training samples and consequent degradation in generation diversity and generalization. We establish, for the first time, a dynamical-systems perspective on this memorization mechanism and prove that regularization is essential to mitigate such degeneration, providing its rigorous theoretical foundation. We propose three novel regularization paradigms: (i) Tikhonov-type, (ii) asymptotic-consistency-based, and (iii) architecture/optimization-induced—arising from network underparameterization, early stopping, and optimization-path constraints. Extensive experiments on unconditional and conditional image generation demonstrate that our methods effectively suppress memorization and restore generalization capability. The work thus delivers both theoretical insights and practical guidelines for robust, interpretable diffusion model training.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Diffusion models have emerged as a powerful framework for generative modeling. At the heart of the methodology is score matching: learning gradients of families of log-densities for noisy versions of the data distribution at different scales. When the loss function adopted in score matching is evaluated using empirical data, rather than the population loss, the minimizer corresponds to the score of a time-dependent Gaussian mixture. However, use of this analytically tractable minimizer leads to data memorization: in both unconditioned and conditioned settings, the generative model returns the training samples. This paper contains an analysis of the dynamical mechanism underlying memorization. The analysis highlights the need for regularization to avoid reproducing the analytically tractable minimizer; and, in so doing, lays the foundations for a principled understanding of how to regularize. Numerical experiments investigate the properties of: (i) Tikhonov regularization; (ii) regularization designed to promote asymptotic consistency; and (iii) regularizations induced by under-parameterization of a neural network or by early stopping when training a neural network. These experiments are evaluated in the context of memorization, and directions for future development of regularization are highlighted.

Problem

• Image Generation
• Diffusion Models
• Data Memorization

Innovation

• Diffusion Models
• Regularization Strategies
• Data Memorization Prevention