SUN: Reaching for Novelty in Reinforcement Learning

2026-09-08

AI Summary

该论文提出了一种名为SUN的方法，通过结合新颖性和可达性来改进强化学习中的探索问题，适用于任何离线策略RL算法。

Abstract

Exploration in reinforcement learning (RL) remains a fundamental challenge. Recent goal-conditioned RL strategies (which select goals to encourage broader state coverage) have shown promising results, but none scores a goal by novelty and reachability jointly: the two signals are traded off by hand, applied in sequence, or one is neglected outright. In this paper, we introduce a reachability-aware goal-selection framework that explicitly integrates these two aspects, and that can be seamlessly incorporated into any off-policy RL algorithm. To this aim, we propose SUccessor-to-Novelty (SUN), an indicator derived from successor value functions to identify goals that are both novel and reachable. We prove that SUN recovers count-based bonuses in the limit, bounds short-horizon hitting probabilities, and provably rejects unreachable goals. We further present an adaptive goal-selection strategy that leverages these properties, and an accurate yet lightweight pseudocount to avoid the overhead of classic methods. We back up all our claims with thorough benchmarks: SUN consistently outperforms state-of-the-art methods in standard and novel environments with unreachable or hard-to-reach states, irreversible transitions, obstacles, mazes, and unbounded spaces.

Problem

• exploration
• reinforcement learning
• goal-conditioned
• novelty
• reachability

Innovation

• reachability-aware goal-selection
• successor value functions
• novel and reachable goals
• adaptive goal-selection strategy
• lightweight pseudocount