Conservation Laws for Diffusion Models

2026-07-10

AI Summary

Diffusion models are typically trained with a denoising objective, which struggles to accurately characterize data likelihood and lacks a unified theoretical framework for both discrete and continuous settings. This work establishes a conservation law for memoryless noise processes based on the generalized extrinsic information transfer (GEXIT) function, expressing the data–model cross-entropy as an integral of information-theoretic derivatives along the noise trajectory. This formulation precisely relates likelihood to local posteriors and reveals the locality of the information-theoretic derivative, thereby unifying likelihood evaluation for both discrete and continuous diffusion models. Experiments on synthetic Markov sources, text8, and CIFAR-10 validate the theoretical predictions, demonstrating that performance differences of denoisers under varying noise levels stem from changes in posterior approximation accuracy.

Abstract

While autoregressive models optimize the exact data likelihood via the chain rule, diffusion models are typically trained with denoising objectives. We develop conservation laws based on generalized extrinsic information transfer (GEXIT) functions for a broad class of memoryless noise processes, showing that the data--model cross-entropy (CE) can be characterized exactly as an integral of local information-theoretic derivatives along the noise path. This yields a unified characterization of the likelihood for discrete and continuous diffusion, with the Gaussian case reducing to the well-known mutual information--minimum mean-square error (I-MMSE) relationship. An immediate implication is a locality property: one can compute the information-theoretic derivatives using only the marginal posteriors along the noise path. As a result, training reduces to learning the marginal posteriors by minimizing the negative log-likelihood. While the conservation law implies that the entropy does not depend on the noise path, finite-capacity denoisers approximate the posteriors with varying accuracy across noise types, leading to differences in performance. We validate these predictions on synthetic Markov sources and standard benchmarks, including text8 and CIFAR-10.

Problem

• diffusion models
• conservation laws
• cross-entropy
• noise processes
• denoisers

Innovation

• conservation laws
• diffusion models
• GEXIT functions
• information-theoretic derivatives
• marginal posteriors