Measuring Semantic Information Production in Generative Diffusion Models

2025-06-12

AI Summary

This study investigates when semantic category decisions emerge during the reverse denoising process of generative diffusion models. Method: We propose an information flow rate metric based on the time derivative of conditional entropy to quantify the dynamic emergence of semantic information; we design an online Bayesian classifier coupled with a conditional entropy estimation framework, and conduct experiments on DDPM using CIFAR-10 and a 1D Gaussian mixture model. Contribution/Results: We find that semantic information flow peaks in the mid-stage of denoising and vanishes toward the final steps; entropy change rates diverge significantly across classes, revealing temporal heterogeneity in semantic decision-making. This work challenges the conventional assumption of uniform temporal semantics and establishes the first interpretable, computationally tractable framework for analyzing the temporal dynamics of semantic decisions in diffusion models—providing a novel theoretical tool for understanding generative mechanisms and enabling controllable generation.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Content-based information diffusion
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

It is well known that semantic and structural features of the generated images emerge at different times during the reverse dynamics of diffusion, a phenomenon that has been connected to physical phase transitions in magnets and other materials. In this paper, we introduce a general information-theoretic approach to measure when these class-semantic"decisions"are made during the generative process. By using an online formula for the optimal Bayesian classifier, we estimate the conditional entropy of the class label given the noisy state. We then determine the time intervals corresponding to the highest information transfer between noisy states and class labels using the time derivative of the conditional entropy. We demonstrate our method on one-dimensional Gaussian mixture models and on DDPM models trained on the CIFAR10 dataset. As expected, we find that the semantic information transfer is highest in the intermediate stages of diffusion while vanishing during the final stages. However, we found sizable differences between the entropy rate profiles of different classes, suggesting that different"semantic decisions"are located at different intermediate times.

Problem

• Measure when class-semantic decisions occur in diffusion models
• Estimate conditional entropy of class labels during generation
• Analyze differences in semantic information transfer across classes

Innovation

• Online Bayesian classifier estimates conditional entropy
• Time derivative identifies peak information transfer intervals
• Class-specific entropy profiles reveal semantic decision timing