Efficient Hierarchical Implicit Flow Q-learning for Offline Goal-conditioned Reinforcement Learning

2026-04-10

AI Summary

This work addresses the limitations of Gaussian policies in offline goal-conditioned reinforcement learning—namely, their restricted expressiveness and the difficulty of high-level policies in generating effective subgoals—by proposing a Goal-Conditioned Mean Flow Policy. The method introduces, for the first time, an average velocity field into a hierarchical offline goal-conditioned reinforcement learning framework, enabling efficient single-step sampling from complex goal distributions through implicit modeling. Additionally, it incorporates a LeJEPA contrastive loss to enhance the discriminability and generalization of goal embeddings. Evaluated on both state-based and pixel-based tasks in the OGBench benchmark, the proposed approach significantly outperforms existing methods, overcoming the performance bottlenecks inherent in conventional Gaussian policies.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Search and Optimization: Learning to Search
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Offline goal-conditioned reinforcement learning (GCRL) is a practical reinforcement learning paradigm that aims to learn goal-conditioned policies from reward-free offline data. Despite recent advances in hierarchical architectures such as HIQL, long-horizon control in offline GCRL remains challenging due to the limited expressiveness of Gaussian policies and the inability of high-level policies to generate effective subgoals. To address these limitations, we propose the goal-conditioned mean flow policy, which introduces an average velocity field into hierarchical policy modeling for offline GCRL. Specifically, the mean flow policy captures complex target distributions for both high-level and low-level policies through a learned average velocity field, enabling efficient action generation via one-step sampling. Furthermore, considering the insufficiency of goal representation, we introduce a LeJEPA loss that repels goal representation embeddings during training, thereby encouraging more discriminative representations and improving generalization. Experimental results show that our method achieves strong performance across both state-based and pixel-based tasks in the OGBench benchmark.

Problem

• offline goal-conditioned reinforcement learning
• hierarchical reinforcement learning
• long-horizon control
• policy expressiveness
• subgoal generation

Innovation

• mean flow policy
• hierarchical reinforcement learning
• offline goal-conditioned RL
• LeJEPA loss
• average velocity field