Understanding Generalization in Diffusion Models via Probability Flow Distance

2025-05-26

AI Summary

Existing diffusion models lack theoretically rigorous and computationally efficient metrics for evaluating generalization capability. Method: We propose the Probability Flow Distance (PFD), a distribution-level generalization metric defined via discrepancies between probability flow ODE trajectories induced by the noise-to-data mapping. PFD is both theoretically interpretable and efficiently computable in high-dimensional image spaces. We establish the first generalization assessment framework grounded in probability flow ODEs, integrating teacher-student protocols, numerical differentiation, and trajectory alignment techniques. Results: Empirically, PFD uncovers three novel generalization phenomena—scaling laws, early learning, and double-descent dynamics—and provides theoretical interpretation via bias-variance decomposition. By unifying theoretical analysis and empirical investigation, PFD furnishes a principled foundation for studying generalization mechanisms in diffusion models.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Diffusion models have emerged as a powerful class of generative models, capable of producing high-quality samples that generalize beyond the training data. However, evaluating this generalization remains challenging: theoretical metrics are often impractical for high-dimensional data, while no practical metrics rigorously measure generalization. In this work, we bridge this gap by introducing probability flow distance ($ exttt{PFD}$), a theoretically grounded and computationally efficient metric to measure distributional generalization. Specifically, $ exttt{PFD}$ quantifies the distance between distributions by comparing their noise-to-data mappings induced by the probability flow ODE. Moreover, by using $ exttt{PFD}$ under a teacher-student evaluation protocol, we empirically uncover several key generalization behaviors in diffusion models, including: (1) scaling behavior from memorization to generalization, (2) early learning and double descent training dynamics, and (3) bias-variance decomposition. Beyond these insights, our work lays a foundation for future empirical and theoretical studies on generalization in diffusion models.

Problem

• Measure generalization in diffusion models efficiently
• Introduce probability flow distance (PFD) metric
• Analyze key generalization behaviors empirically

Innovation

• Introducing probability flow distance (PFD) metric
• Using PFD for teacher-student evaluation protocol
• Analyzing generalization behaviors in diffusion models