Reversal Q-Learning

2026-06-16

AI Summary

This work addresses the challenges in offline reinforcement learning where flow-based policy training is prone to bias and variance and struggles with long-horizon tasks. The authors propose a novel approach grounded in an extended Markov decision process framework, which models flow refinement steps as independent actions and leverages backward generation of synthetic on-policy trajectories to enable efficient offline learning. By integrating a dedicated bias-variance reduction mechanism, the method trains full-capacity flow policies directly without requiring backpropagation through time, thereby making more effective use of value function information. Evaluated on 50 challenging simulated robotic tasks, the proposed method significantly outperforms existing flow-based offline reinforcement learning approaches, achieving state-of-the-art average performance.

Abstract

Iterative generative modeling techniques, such as flow matching, provide powerful tools to model complex behaviors for effective offline reinforcement learning (RL). In this work, we propose a new off-policy RL algorithm that trains a flow policy based on prior data. Our idea starts from the "expanded" Markov decision process (MDP) framework, which treats individual flow refinement steps as separate actions in an MDP. To enable off-policy RL within this framework, we apply two techniques: we generate virtual on-policy trajectories (by "reversing" flows) to make this framework compatible with prior data, and we apply a bias-and-variance reduction technique to mitigate the curse of horizon in off-policy RL. We call the resulting algorithm Reversal Q-learning (RQL). RQL has several advantages over previous flow-based RL methods: it does not suffer from backpropagation through time, makes better use of the learned value function, and directly trains the full, expressive flow policy. Through our experiments on 50 challenging simulated robotic tasks, we show that RQL leads to the best average offline RL performance compared to state-of-the-art flow-based offline RL algorithms.

Problem

• offline reinforcement learning
• flow-based policy
• off-policy RL
• Markov decision process
• flow matching

Innovation

• Reversal Q-learning
• flow-based reinforcement learning
• off-policy RL
• expanded MDP
• bias-variance reduction