Diffusion-APO: Trajectory-Aware Direct Preference Alignment for Video Diffusion Transformers

2026-05-08

AI Summary

This work addresses the challenge of aligning large-scale video diffusion models with human intent, a task hindered by biased reward models and suboptimal timestep sampling that cause mismatches between training and inference trajectories. The authors propose Diffusion-APO, a trajectory-aware preference alignment algorithm that achieves direct preference optimization without relying on scalar reward models. By jointly modeling the training noise distribution and the inference denoising path, Diffusion-APO enhances gradient effectiveness. The method introduces a modular RLHF framework integrating online ranking, semi-online anchoring, offline refinement, and distillation-aware drift correction, enabling flexible multi-stage alignment. Experiments demonstrate that Diffusion-APO significantly outperforms baselines in both visual quality and instruction following, while preserving generation fidelity under model acceleration, offering an end-to-end solution for scalable video diffusion alignment.

Abstract

Efficiently aligning large-scale video diffusion models with human intent requires a scalable and trajectory-aware pathway that bridges the inherent discrepancy between training noise distributions and practical inference trajectories. While existing paradigms such as Direct Preference Optimization (DPO) and Group Relative Policy Optimization (GRPO) attempt to address this, they are often hindered by either reliance on bias-prone, complex reward models or suboptimal timestep sampling. In this paper, we propose Diffusion-APO (Aligned Preference Optimization), a trajectory-aware algorithm that resolves this misalignment by synchronizing training noise with inference-time denoising paths to maximize gradient signal efficacy. To translate this algorithmic innovation into a practical solution, we introduce a unified and modular RLHF framework that integrates online ranking, half-online anchoring, offline refinement, and distillation-aware drift correction. This framework enables flexible, multi-stage preference alignment across diverse data and computational constraints without relying on scalar-reward-based policy gradients. Through extensive experiments, we demonstrate that Diffusion-APO consistently outperforms standard baselines in visual quality and instruction following, while effectively preserving generative fidelity during model acceleration, providing a robust, end-to-end pathway for scalable video diffusion alignment.

Problem

• video diffusion models
• preference alignment
• trajectory-aware
• human intent alignment
• inference trajectories

Innovation

• trajectory-aware alignment
• video diffusion transformers
• preference optimization
• reward-free RLHF
• denoising path synchronization