Abstraction for Offline Goal-Conditioned Reinforcement Learning

2026-05-21

AI Summary

Offline goal-conditioned reinforcement learning often suffers from low sample efficiency due to redundancy in state-goal pairs. This work proposes a hierarchical policy that abstracts goal conditioning away from an absolute reference frame by introducing relativized options and multi-level state representations, enabling experience reuse across similar contexts. Unlike conventional hierarchical approaches that primarily provide temporal abstraction, the proposed architecture explicitly exploits hierarchy to eliminate redundancy in the state-goal space. Experimental results demonstrate that the method substantially improves performance on offline goal-conditioned tasks, validating the effectiveness of the introduced abstract inductive bias.

Abstract

Markov Decision Processes (MDPs) often exhibit significant redundancy due to symmetries and shared structure across state-goal pairs in real-world Goal-Conditioned Reinforcement Learning (GCRL). While hierarchical policies have been motivated for horizon reduction via temporal abstraction in offline GCRL, we demonstrate that hierarchy also enables absolute abstraction. By introducing relativised options as well as distinct representations for different levels of the hierarchy, we demonstrate how an agent can reuse experience across similar contexts of the state-space. Based on this framework, we introduce two simple algorithms for learning relativised options and abstracting from the absolute frame of reference. Our experiments show that such inductive biases significantly improve performance in offline GCRL.

Problem

• Goal-Conditioned Reinforcement Learning
• Offline Reinforcement Learning
• Abstraction
• Redundancy
• Markov Decision Processes

Innovation

• relativised options
• absolute abstraction
• hierarchical reinforcement learning
• offline goal-conditioned RL
• inductive bias