Learn to Guide Your Diffusion Model

2025-10-01

AI Summary

To address the distribution shift and degraded semantic alignment caused by static, large guidance weights in classifier-free guidance (CFG), this paper proposes dynamic CFG: a continuously differentiable guidance weight function learned end-to-end, conditioned on both the input context and the denoising start/target timestep. The function is optimized by minimizing the discrepancy between the conditional distribution and the true data distribution, and inherently supports reward-based guidance (e.g., CLIP Score) for downstream-targeted generation distribution control. Integrated into standard diffusion frameworks, dynamic CFG requires only joint training of a lightweight weight network without modifying the backbone model. Experiments on image generation and text-to-image synthesis demonstrate significant FID reduction and improved prompt–image alignment, validating its dual advantages in perceptual quality and distribution fidelity.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Classifier-free guidance (CFG) is a widely used technique for improving the perceptual quality of samples from conditional diffusion models. It operates by linearly combining conditional and unconditional score estimates using a guidance weight $ω$. While a large, static weight can markedly improve visual results, this often comes at the cost of poorer distributional alignment. In order to better approximate the target conditional distribution, we instead learn guidance weights $ω_{c,(s,t)}$, which are continuous functions of the conditioning $c$, the time $t$ from which we denoise, and the time $s$ towards which we denoise. We achieve this by minimizing the distributional mismatch between noised samples from the true conditional distribution and samples from the guided diffusion process. We extend our framework to reward guided sampling, enabling the model to target distributions tilted by a reward function $R(x_0,c)$, defined on clean data and a conditioning $c$. We demonstrate the effectiveness of our methodology on low-dimensional toy examples and high-dimensional image settings, where we observe improvements in Fréchet inception distance (FID) for image generation. In text-to-image applications, we observe that employing a reward function given by the CLIP score leads to guidance weights that improve image-prompt alignment.

Problem

• Learning dynamic guidance weights for diffusion models
• Improving distributional alignment in guided sampling
• Optimizing conditional generation with reward functions

Innovation

• Learned guidance weights vary with conditioning and time
• Minimizes distributional mismatch between true and guided samples
• Extends framework to reward-guided sampling for improved alignment