ReFPO: Reflow Regularization for Flow Matching Policy Gradients

2026-06-19

AI Summary

This work addresses the instability, sensitivity to discretization, and inefficiency in multi-step inference that plague Flow Policy Optimization (FPO) methods. To overcome these limitations, the authors propose an explicit Reflow geometric regularization technique that seamlessly integrates advantage-weighted updates with path correction. The approach uncovers the implicit Reflow structure underlying FPO gradient updates and implements effective regularization via a single line of code, incurring no additional computational overhead. Empirical evaluations on GridWorld, MuJoCo Playground, and high-dimensional Humanoid tasks demonstrate that the method substantially enhances training stability and robustness to discretization, consistently outperforming baseline approaches in average performance while achieving single-step inference results that match or exceed those of conventional multi-step methods.

Abstract

We present Reflow-regularized Flow Matching Policy Gradients (ReFPO), a simple online RL method that adds explicit Reflow regularization to FPO for efficient flow-based control. We uncover a key structural property: the gradient updates in Flow Matching Policy Gradients (FPO) can be interpreted as an implicit advantage-weighted Reflow process, providing a new geometric perspective on flow-based policy gradients. Building on this insight, ReFPO introduces an explicit geometric regularizer that can be implemented with a single line of code change without incurring additional computational overhead or auxiliary distillation stages. By synergizing advantage-guided updates with path rectification, our method reduces CFM proxy-ratio spikes, stabilizes PPO-style training, and enables high-fidelity one-step inference that often matches or exceeds multi-step performance. We experimentally demonstrate that ReFPO improves average performance and discretization robustness across GridWorld, MuJoCo Playground, and high-dimensional Humanoid Control tasks, providing a scalable and stable approach for generative policies in complex physical simulations.

Problem

• flow matching
• policy gradients
• reflow regularization
• training stability
• discretization robustness

Innovation

• Reflow regularization
• Flow Matching Policy Gradients
• advantage-weighted updates
• geometric regularizer
• one-step inference