Offline Preference Optimization for Rectified Flow with Noise-Tracked Pairs

2026-05-10

AI Summary

This work addresses a critical limitation in existing text-to-image preference datasets, which only record final win/lose image outcomes and thus cannot support rectified flow models that rely on specific prior noise for generation trajectories. To overcome this, the authors propose the PNAPO framework, which introduces paired prior noise into preference data for the first time, enabling a noise-aware offline preference optimization approach. Leveraging the straight-line trajectory property of rectified flows, the method performs interpolation of intermediate states and incorporates a dynamic regularization strategy, significantly reducing trajectory estimation variance and improving sample efficiency. Experiments demonstrate that PNAPO effectively enhances preference alignment in state-of-the-art rectified flow-based text-to-image models while substantially lowering computational costs during training.

Abstract

Existing preference datasets for text-to-image models typically store only the final winner/loser images. This representation is insufficient for rectified flow (RF) models, whose generation is naturally indexed by a specific prior noise sample and follows a nearly straight denoising trajectory. In contrast, prior DPO-style alignment for diffusion models commonly estimates trajectories using an independent forward noising process, which can be mismatched to the true reverse dynamics and introduces unnecessary variance. We propose Prior Noise-Aware Preference Optimization (PNAPO), an off-policy alignment framework specialized for rectified flow. PNAPO augments preference data by retaining the paired prior noises used to generate each winner/loser image, turning the standard (prompt, winner, loser) triplet into a sextuple. Leveraging the straight-line property of RF, we estimate intermediate states via noise-image interpolation, which constrains the trajectory estimation space and yields a tighter surrogate objective for preference optimization. In addition, we introduce a dynamic regularization strategy that adapts the DPO regularization based on (i) the reward gap between winner and loser and (ii) training progress, improving stability and sample efficiency. Experiments on state-of-the-art RF T2I backbones show that PNAPO consistently improves preference metrics while substantially reducing training compute.

Problem

• preference optimization
• rectified flow
• noise-tracked pairs
• text-to-image generation
• trajectory estimation

Innovation

• Rectified Flow
• Preference Optimization
• Noise-Aware Alignment
• Dynamic Regularization
• Offline Reinforcement Learning