🤖 AI Summary
Standard score matching in diffusion models overlooks the structural properties of the forward noising process. Method: We propose a novel score learning framework based on spectral decomposition of the Markov operator governing the forward process. Leveraging the analytical tractability of the Markovian noising dynamics, we derive explicit forward solutions and operator-driven gradient estimators. We introduce— for the first time—the incorporation of Markov operator structure into score matching, yielding two key techniques: (i) diffusion kernel smoothing on Riemannian manifolds to reduce reliance on neural network parametrization, and (ii) variance-reduced score matching under operator constraints. Parameter estimation is achieved via time-dependent kernel mean embeddings, requiring only sample means. Results: Experiments demonstrate significantly reduced dependence on neural score approximators in low-dimensional settings, and consistently improved score estimation accuracy and training convergence stability across both low- and high-dimensional tasks.
📝 Abstract
Diffusion models are typically trained using score matching, yet score matching is agnostic to the particular forward process that defines the model. This paper argues that Markov diffusion models enjoy an advantage over other types of diffusion model, as their associated operators can be exploited to improve the training process. In particular, (i) there exists an explicit formal solution to the forward process as a sequence of time-dependent kernel mean embeddings; and (ii) the derivation of score-matching and related estimators can be streamlined. Building upon (i), we propose Riemannian diffusion kernel smoothing, which ameliorates the need for neural score approximation, at least in the low-dimensional context; Building upon (ii), we propose operator-informed score matching, a variance reduction technique that is straightforward to implement in both low- and high-dimensional diffusion modeling and is demonstrated to improve score matching in an empirical proof-of-concept.