D2PO: Optimizing Diffusion Samplers via Dynamic Preference

2026-07-07

AI Summary

This work addresses the degradation of high-frequency textures in low-NFE diffusion samplers when distilling knowledge from high-NFE teacher models, which leads to generation quality misaligned with human perception. To mitigate this issue, the authors reformulate sampling strategy optimization as a preference alignment problem and introduce a dynamic preference optimization framework. Specifically, they model the sampling process as an energy-based model and integrate a pretrained score network with Direct Preference Optimization (DPO) to construct self-improving preference signals during training. The proposed method substantially outperforms conventional regression-based baselines under low-NFE constraints, achieving better alignment with human-perceived image quality and effectively unlocking the potential of high-quality teacher models.

Abstract

We propose D2PO (Dynamic Direct Preference Optimization), a principled framework for optimizing diffusion sampling policies with respect to timestep schedules and classifier-free guidance (CFG) weights. Our work is motivated by a fundamental limitation of existing student-teacher regression frameworks; low-NFE student samplers are trained to mimic high-NFEteachers, often sacrificing high-frequency texture fidelity while preserving coarse global structures, thereby misaligning the sampler with perceptual quality. D2PO addresses this challenge by reformulating sampler optimization as a preference-based alignment problem, leveraging the Direct Preference Optimization (DPO) framework. To make DPO applicable to diffusion samplers, we model the sampling policy as an energy-based model (EBM), transforming preference comparisons into tractable energy differences. We further introduce a novel energy formulation derived directly from the pretrained score network, enabling preference evaluation in perturbed spaces that jointly capture structural consistency and fine-grained details. Moreover, we introduce dynamic preferences, where the preferred samples used for alignment progressively improve as the sampling policies are learned. This self-improving mechanism replaces rigid static teacher supervision with an iterative, preference-guided refinement process, providing progressively stronger alignment signals. Extensive experiments demonstrate that D2PO aligns diffusion samplers with perceptual quality more faithfully, unlocking the full potential of high-quality teachers and consistently outperforming conventional regression-based schedulers under low-NFE constraints.

Problem

• diffusion samplers
• perceptual quality
• low-NFE
• texture fidelity
• student-teacher regression

Innovation

• Dynamic Preference Optimization
• Diffusion Sampling
• Energy-Based Model
• Perceptual Alignment
• Classifier-Free Guidance