Tightening the Score Matching Gap for Diffusion Models

2026-07-05

AI Summary

This work addresses the loose theoretical gap between score-matching loss and the quality of generated samples in diffusion models, which hinders accurate evaluation of generative performance. By integrating entropy dissipation, logarithmic Sobolev inequalities, and reflection coupling techniques with ergodicity theory for Langevin diffusions, the authors leverage the contractive properties of the reverse process to derive substantially tighter error bounds for the KL divergence, reverse KL divergence, and Wasserstein distance. The analysis reveals that the regularity of score estimation at low noise scales plays a pivotal role in narrowing this gap, thereby elucidating how the fidelity of score approximation governs generative performance. This study advances the theoretical understanding of training dynamics and evaluation metrics in diffusion models.

Abstract

Diffusion models (DMs) are a state-of-the-art generative method to approximately sample from an unknown distribution. Their training and evaluation primarily rely on an Evidence Lower Bound (ELBO), which relates the Kullback-Leibler (KL) divergence of model samples to the score matching loss along the path, which serves as a tractable surrogate. The difference between sample quality and the score matching loss produced by this bound leads to the \emph{score matching gap}, which is known to be tight in the worst-case but not descriptive of sample quality in general. In this work, we provide a theoretical analysis of this gap, developing tighter bounds for three metrics: KL divergence, reverse KL divergence, and Wasserstein distance, effectively exploiting the regularity of the class of score estimators. Our results suggest that the quality of the score approximation has more impact on closing the score matching gap for low noise scales. To obtain these bounds, our key technical insight is to exploit the contraction properties of the backward processes. In particular, we rely on entropy flows, logarithmic Sobolev inequalities and reflection couplings, rigorously linking the ergodicity of the Langevin diffusion to the score matching gap problem.

Problem

• score matching gap
• diffusion models
• sample quality
• KL divergence
• Wasserstein distance

Innovation

• score matching gap
• diffusion models
• contraction properties
• logarithmic Sobolev inequalities
• entropy flows