Direction-Conditioned Policies via Compositional Subgoal Scoring for Online Goal-Conditioned Reinforcement Learning

2026-06-15

AI Summary

This work addresses the inefficiency of online goal-conditioned reinforcement learning when goals lie outside the data distribution, stemming from a lack of geometric structure. The authors propose a directional conditional policy framework that decomposes goal reaching into subgoal scoring and a direction–distance conditioned policy, both sharing an InfoNCE-based contrastive representation. During training, visited states aligned with the final goal are selected as subgoals; at deployment, the target directly replaces the subgoal. Leveraging Hamilton–Jacobi–Bellman theory, they prove that optimal actions depend solely on the gradient of the value function and characterize a controllable subspace where directional conditioning fails, thereby decoupling training from deployment. Experiments across nine environments show consistent improvements over Contrastive RL, particularly in manipulation and obstacle-interaction tasks, with the learned distance exhibiting quasi-metric topology and failure cases aligning with theoretical predictions.

Abstract

Hamilton-Jacobi-Bellman theory implies that the optimal goal-conditioned action depends on the goal only through the gradient of the goal-reaching distance at the current state, yet standard online GCRL still conditions the actor on the raw goal -- a signal that is geometrically uninformative when the goal is far from the data distribution. We propose Direction-Conditioned Policies (DCP), a fully online method that decomposes goal-reaching into two components sharing one InfoNCE representation $ψ$: a subgoal-scoring step that selects a visited state $z_t$ aligned with the final goal $g$ in $ψ_g$, and a direction-conditioned actor that consumes the unit direction $d_t$ and magnitude $r_t$ from $ψ(s_t)$ to $ψ(z_t)$. The two components train jointly, factor cleanly at deployment (subgoal scoring is removed, while direction conditioning remains with $g$ in place of $z_t$), and admit independent modification at the same $(d_t,r_t)$ interface. We prove three results. First, direction sufficiency under HJB: the optimal action under control-affine dynamics depends on the goal only through the value gradient. Second, a quantitative bound showing that, under mild conditions on the learned representation and assuming the scoring rule returns an on-path $z_t$, the actor's conditioning input at training and at deployment coincide up to representation error and geodesic slack. Third, a controllable-subspace characterization of when directional conditioning fails. Across nine environments, DCP improves over Contrastive RL on most final metrics, with the largest gains on manipulation and obstacle-interaction tasks; a qualitative analysis of the learned $ψ$-distance landscape shows the contrastive representation behaves as an online quasimetric encoding environment topology, and the single failure case (AntSoccer) localizes to a learned-gradient pathology that the theory anticipates.

Problem

• goal-conditioned reinforcement learning
• online learning
• geometric information
• value gradient
• representation learning

Innovation

• Direction-Conditioned Policies
• Goal-Conditioned Reinforcement Learning
• Compositional Subgoal Scoring
• InfoNCE Representation
• Hamilton-Jacobi-Bellman Theory