Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learning

2026-05-27

AI Summary

This work addresses the challenges of sparse supervision and accumulated bootstrapping errors in long-horizon offline goal-conditioned reinforcement learning, where distant state-goal pairs provide weak learning signals. To overcome these issues, the authors propose the CFHRL framework, which employs a learnable reachability-cost-driven adaptive recursion mechanism to dynamically refine distant goals and assess the local executability of subgoals. This enables flexible adjustment of hierarchical depth without relying on a fixed hierarchy. The method only requires subgoals to yield reliable progress rather than global optimality and leverages candidate goals sampled from the replay buffer for training. Evaluated on multiple long-horizon tasks in OGBench, CFHRL significantly outperforms baseline methods, and ablation studies confirm the effectiveness of its goal refinement and stopping mechanisms.

Abstract

Offline goal-conditioned reinforcement learning (GCRL) is challenging in long-horizon tasks, where distant state--goal pairs provide weak supervision and value estimates become vulnerable to accumulated bootstrapping errors. Hierarchical methods mitigate this difficulty by introducing intermediate subgoals, but fixed temporal abstractions or fixed hierarchy depths can be mismatched to state--goal pairs with different reachability horizons. We propose Coarse-to-Fine Hierarchical Goal Reinforcement Learning (CFHRL), a fully offline GCRL framework that adaptively refines distant goals before execution. Starting from the final goal, CFHRL recursively proposes intermediate targets, trained from replay-supported candidates, and stops refinement once the current target is estimated to be locally executable by a learned reachability cost. The key idea is that a subgoal need not be an exact midpoint or globally optimal waypoint; it only needs to provide reliable progress and reduce the remaining reaching difficulty, enabling subsequent refinement over shorter horizons. A stylized analysis further supports the robustness of approximate recursive contraction. Experiments on OGBench show substantial gains on several long-horizon tasks, with ablations validating the proposed refinement and stopping mechanisms

Problem

• offline goal-conditioned reinforcement learning
• long-horizon tasks
• subgoal refinement
• bootstrapping errors
• hierarchical reinforcement learning

Innovation

• adaptive subgoal refinement
• offline goal-conditioned reinforcement learning
• hierarchical reinforcement learning
• reachability cost
• long-horizon tasks