Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning

2026-05-02

AI Summary

This work addresses the high computational cost and low sample efficiency commonly associated with expressive flow policies and distributional value functions in offline reinforcement learning. The authors propose FAN, an algorithm that uniquely integrates single-step flow policy iteration with distributional value estimation using only a single Gaussian noise sample, while incorporating behavioral regularization. This approach preserves policy expressiveness yet significantly enhances both training and inference efficiency. Theoretical analysis establishes the convergence and performance advantages of the method. Empirical results demonstrate that FAN achieves state-of-the-art performance on robotic manipulation and locomotion tasks while substantially reducing computational overhead.

Abstract

We propose Flow-Anchored Noise-conditioned Q-Learning (FAN), a highly efficient and high-performing offline reinforcement learning (RL) algorithm. Recent work has shown that expressive flow policies and distributional critics improve offline RL performance, but at a high computational cost. Specifically, flow policies require iterative sampling to produce a single action, and distributional critics require computation over multiple samples (e.g., quantiles) to estimate value. To address these inefficiencies while maintaining high performance, we introduce FAN. Our method employs a behavior regularization technique that utilizes only a single flow policy iteration and requires only a single Gaussian noise sample for distributional critics. Our theoretical analysis of convergence and performance bounds demonstrates that these simplifications not only improve efficiency but also lead to superior task performance. Experiments on robotic manipulation and locomotion tasks demonstrate that FAN achieves state-of-the-art performance while significantly reducing both training and inference runtimes. We release our code at https://github.com/brianlsy98/FAN.

Problem

• offline reinforcement learning
• flow policies
• distributional critics
• computational efficiency
• expressive policies

Innovation

• offline reinforcement learning
• flow policy
• distributional critic
• behavior regularization
• computational efficiency