The Reward Was in Your Data All Along: Correcting Flow Matching with Discriminator-Guided RL

2026-06-17

AI Summary

This work addresses the structural mismatch between the L2 regression loss used during training of flow matching models and the perceptual quality required at inference. To bridge this gap, the authors propose Discriminator-Guided Reinforcement Learning (DRL), which trains a discriminator in a pretrained representation space to distinguish real from generated samples. The discriminator’s logits are interpreted as the log-likelihood ratio between data and model distributions, serving as an optimal reward signal that requires no human preference data. Combined with KL regularization for policy optimization, DRL significantly improves generation quality across flow matching architectures such as SiT and JiT: unconditional FID drops from 9.38 to 2.62, DINOv3 semantic-space Fréchet distance decreases from 88.2 to 19.3, and human preference scores increase while artifacts like oversaturation are mitigated.

Abstract

Score- and flow-matching models often rely on preference-based reinforcement learning for two purposes: aligning with subjective preferences and, surprisingly, recovering properties such as visual realism and coherent object structure that matching-based training is intended to learn from the data itself. We argue that this reflects a structural mismatch. Matching losses measure $\ell_2$ regression error on the velocity or score field under training-time marginals, a proxy poorly aligned with the visual and semantic properties that determine sample quality at inference. Given a reward aligned with these properties, RL sidesteps the mismatch by evaluating the model on its own samples and following the reward landscape directly. The challenge is to obtain such a reward without relying on human preferences, which are expensive and conflate data realism with annotator inclinations. We propose Discriminator-Guided RL (DRL). DRL trains a discriminator to separate data from base-model samples in a pretrained representation space and uses its logit as the reward in KL-regularized RL. The pretrained space restricts the discriminator to perceptually meaningful directions, and the logit estimates the log-likelihood ratio between data and model, which is the optimal reward for targeting the data distribution. Across SiT, JiT, REPA, and RAE, DRL reduces guidance-free FID (e.g., $9.38 \to 2.62$ on SiT) and semantic-space FD (e.g., $88.2 \to 19.3$ on DINOv3 for SiT), with consistent gains across all backbones, and improves human-preference rewards without training on them. It also yields a better Pareto frontier between preference reward and image fidelity under subsequent preference-based post-training, increasing alignment while reducing low-level artifacts such as oversaturation and excessive brightness.

Problem

• flow matching
• reinforcement learning
• reward design
• generative modeling
• distribution alignment

Innovation

• Discriminator-Guided RL
• Flow Matching
• Reward Learning
• Generative Modeling
• Perceptual Alignment