FlowDPG: Deterministic Policy Gradient on Flow Matching Policies for Real-World Manipulation

2026-06-20

AI Summary

This work addresses the computational burden and numerical instability of flow-matching–based reinforcement learning in real-world robotic manipulation, which stems from its reliance on backpropagation through time (BPTT). To overcome this limitation, the authors propose FlowDPG, a BPTT-free DDPG-style algorithm that distills critic gradients into a velocity field, thereby integrating demonstration-driven motion with critic-guided corrections during policy optimization. Theoretical analysis demonstrates that FlowDPG’s update direction aligns with the classical deterministic policy gradient. Evaluated on a multi-stage dual-arm AirPods assembly task, FlowDPG achieves a 92% end-to-end success rate, substantially outperforming existing approaches based on value conditioning, auxiliary module adaptation, and adjoint gradients.

Abstract

Real-world reinforcement learning for robotic manipulation remains challenging, and this difficulty is amplified for flow matching policies: applying policy gradient methods to these policies is fundamentally limited by the need to backpropagate through time(BPTT) along the multi-step ODE that maps noise to actions, which is computationally prohibitive and numerically fragile. We propose FlowDPG, a DDPG-style method specifically designed for flow matching policies that distills the critic gradient into the velocity field at training time, bypassing BPTT entirely. Intuitively, FlowDPG combines two complementary vectors: the demonstration-driven velocity that keeps the action feasible, and the critic-driven correction that steers it toward higher value. Our contributions are threefold: (1) a BPTT-free distillation framework that enables stable DDPG-style policy improvement on flow matching policies, (2) a formal connection between the FlowDPG update direction and vanilla Deterministic Policy Gradient via three explicit approximations, and (3) real-world validation on a long-horizon, multi-stage, dual-arm AirPods assembly task, where FlowDPG attains a 92% end-to-end success rate, substantially outperforming recent RL methods spanning value-conditioning, auxiliary-module adaptation, and adjoint-based critic-gradient approaches. Videos and more results are provided on the project page https://flowdpg.github.io.

Problem

• flow matching
• policy gradient
• backpropagation through time
• robotic manipulation
• real-world reinforcement learning

Innovation

• Flow Matching
• Deterministic Policy Gradient
• Backpropagation Through Time (BPTT)
• Velocity Field Distillation
• Real-World Robotic Manipulation