DreOPD: Degraded-Reference Extrapolative On-Policy Distillation for Flow-matching Models

2026-08-10

AI Summary

This work addresses the multi-objective conflicts in post-training of flow matching models and the high-variance gradients and cross-task interference inherent in reinforcement learning–based trajectory optimization. To this end, the authors propose a novel approach that integrates reward extrapolation with policy distillation. By reformulating implicit reward extrapolation as a closed-form velocity regression and introducing a mildly degraded reference signal to strengthen teacher–reference contrast, the method explicitly defines the extrapolation direction. Notably, it is the first to embed an extrapolation mechanism within a policy-based distillation framework. Combining online distillation with a multi-teacher setup, the proposed method consistently outperforms conventional OPD and multitask reinforcement learning baselines in both single- and multi-teacher settings, and surpasses specialized teacher models across most evaluation metrics.

Abstract

Flow-matching models are now a mainstream method to image generation, but its adaptation to diverse downstream scenarios typically relies on post-training, which may cause conflicts among task-specific optimization objectives. Reinforcement learning enables direct optimization of task-specific rewards beyond the original models, yet trajectory-level optimization may incur high-variance gradients and cross-task interference. On-policy distillation (OPD) offers dense and stable supervision on student rollouts, but conventional teacher matching remains imitation-based. We propose DreOPD, a Degraded-reference extrapolative OPD method for flow-matching models that bridges these two paradigms. Our DreOPD converts implicit reward extrapolation into closed-form velocity regression, enabling extrapolative post-training with the stability of OPD. It further uses a mildly degraded reference to strengthen the teacher-reference contrast, yielding a clearer extrapolation direction. Experiments on single- and multi-teacher settings show that DreOPD outperforms OPD and multi-task RL baselines in average performance, while surpassing specialized teachers on most metrics.

Problem

• flow-matching models
• post-training
• task-specific optimization
• cross-task interference
• high-variance gradients

Innovation

• flow-matching
• on-policy distillation
• reward extrapolation
• degraded reference
• velocity regression