NFTR: From Provable Mode-Averaging to Geodesic Subgoal Selection in Offline Goal-Conditioned RL

2026-07-08

AI Summary

This work addresses the optimistic bias and Gaussian mode collapse inherent in subgoal selection within offline goal-conditioned reinforcement learning. To overcome these limitations, the authors propose a novel approach that integrates conditional normalizing flows with a triangle inequality–driven relaxed reweighting mechanism. Conditional normalizing flows enable flexible modeling of multimodal subgoal distributions, circumventing the restrictive Gaussian assumption that leads to mode collapse. Furthermore, the method employs an advantage-weighted regression (AWR) framework augmented with triangle-based relaxation scores to stably select subgoals that are both reachable and composable. Theoretical analysis establishes guarantees for monotonic policy improvement and controllable error decomposition. Empirical results demonstrate that the proposed method significantly enhances subgoal reachability and overall policy performance in offline settings.

Abstract

Hierarchical Implicit Q-Learning (HIQL), an offline goal-conditioned RL method, selects subgoals by value-function advantages alone. This rule has two coupled failure modes. Optimistic bias treats lucky stochastic outcomes as skillful choices, and mode collapse reduces a multi-modal subgoal distribution to a single Gaussian mean that often falls in unreachable regions. We propose NFTR (Normalizing Flows subgoal policies with Triangle-slack Reweighting). A conditional Normalizing Flow replaces the Gaussian policy, and a closed-form mode-averaging result identifies NFs as the minimal generative class for AWR-based subgoal selection. A triangle slack score, built on the architectural triangle inequality without relying on distance accuracy, multiplicatively corrects the AWR weight to downweight subgoals whose detour cost exceeds average reachability. Triangle-slack vanishes on geodesics in deterministic MDPs and remains a conservative upper bound on composability violation under stochastic dynamics. The RWDR objective preserves AWR's population-level monotonic improvement and admits a three-term suboptimality decomposition. Together, these two ingredients yield subgoal selection that provably avoids the Gaussian collapse described above and remains stable under stochastic dynamics. GitHub page: https://github.com/erdemtbao/NFTR

Problem

• offline goal-conditioned RL
• subgoal selection
• mode collapse
• optimistic bias
• multi-modal distribution

Innovation

• Normalizing Flows
• Triangle-slack Reweighting
• Offline Goal-Conditioned RL
• Subgoal Selection
• Mode Collapse