Reward-free Pretraining for Reinforcement Learning via Occupancy Coverage Maximization

2026-06-19

AI Summary

This work addresses the challenge of insufficient exploration signals in reinforcement learning under sparse reward settings by proposing a reward-free pretraining method that learns transferable exploration policies through maximizing the entropy of state visitation measures. The key innovation lies in introducing a virtual “sink” state to balance exploration between known and unknown regions, thereby preventing the policy from falling into cyclic expansion–collapse dynamics during training. Built upon an operator-based reinforcement learning framework, the approach leverages a resolvent world model to directly estimate the state visitation measure, circumventing the difficulties associated with conventional density and entropy estimation. Experiments demonstrate that the method significantly improves the uniformity of state coverage in both tabular and pixel-based sparse navigation tasks and provides superior initialization policies for downstream tasks.

Abstract

Sparse rewards pose a central challenge in reinforcement learning, since agents receive no informative signal until they reach their goal. Intrinsic-reward methods address this issue by optimizing non-stationary objectives such as novelty, prediction error, or skill diversity, thereby injecting a supervision signal into the problem. While effective, these methods often require that the extrinsic (sparse) reward can be evaluated -- either online or during offline relabeling of the stored transitions. This limitation is particularly vexing for multi-task, meta-, and continual reinforcement learning, where agents' interactions with the environment are usually reward-free. In this work, we present a method to pre-train transferable exploration policies that rapidly adapt to sparse rewards at downstream task time. Our objective maximizes state-space covering for the occupancy measure, and can be framed in terms of entropy maximization. Its algorithmic implementation, ROVER, leverages recent advances on the operatorial formulation of RL to estimate occupancy with a learned resolvent world model, bypassing common hurdles associated with density and entropy estimation. ROVER further introduces a virtual "sink" state for unexplored regions, balancing coverage of known states with expansion into unseen ones and preventing cyclic expansion-collapse behavior during learning. In tabular and pixel-based sparse navigation tasks, ROVER produces more uniform aggregate coverage and stronger initializations for downstream tasks than standard reward-free baselines.

Problem

• sparse rewards
• reward-free reinforcement learning
• intrinsic reward
• multi-task RL
• exploration

Innovation

• occupancy coverage maximization
• reward-free pretraining
• resolvent world model
• entropy maximization
• exploration policy