Refinement-based Flow Policy Optimization

2026-09-14

AI Summary

本文提出RFPO方法，通过Q函数引导样本优化和自目标流匹配交替训练流策略，解决了在线强化学习中难以直接从隐式定义的分布采样问题。

Abstract

Flow-based policies offer an expressive representation for online reinforcement learning, but conventional flow matching requires samples drawn from the distribution to be modeled. This poses a challenge when the desired action distribution is defined only implicitly by a Q-function, since directly sampling actions from the resulting distribution is generally intractable. We propose Refinement-Based Flow Policy Optimization (RFPO), a novel framework for training a flow policy in online reinforcement learning by alternating between Q-guided sample refinement and self-target flow matching. RFPO first generates actions from Gaussian noise using the current flow policy and then uses a finite-step stochastic refinement procedure to move them toward an energy-based distribution induced by the Q-function. Each refined action is then paired with its corresponding initial noise sample and used as a fixed target for flow-matching training. By repeatedly refining its own outputs and learning from the resulting targets, RFPO incorporates Q-guidance into the policy without requiring direct samples from the target distribution, while retaining the capacity to represent multiple action modes. We further provide a theoretical analysis of the distributional dynamics induced by RFPO. Across six continuous-control tasks, RFPO matches or outperforms a standard Gaussian-policy baseline on almost every task. Experiments on six synthetic two-dimensional target distributions with diverse geometries demonstrate that RFPO captures complex multimodal structure without mode collapse.

Problem

• Flow-based Policies
• Online Reinforcement Learning
• Q-function
• Sample Refinement
• Action Distribution

Innovation

• Refinement-Based Flow Policy Optimization
• Q-guided sample refinement
• self-target flow matching
• Gaussian noise
• multimodal structure