SVL: Goal-Conditioned Reinforcement Learning as Survival Learning

2026-04-19

AI Summary

This work addresses the instability and low sample efficiency of temporal difference (TD) methods in goal-conditioned reinforcement learning by proposing Survival Value Learning (SVL), a framework that reformulates the task as a survival analysis problem. SVL models the distribution of first-hitting times to the goal from any state and establishes a closed-form relationship between value functions and survival probabilities. The approach trains a risk model via maximum likelihood estimation, naturally unifying observed events and right-censored trajectories. It further introduces three practical value estimators—featuring a finite-horizon truncation and two infinite-horizon binning approximations. Evaluated on offline goal-conditioned RL benchmarks, SVL combined with hierarchical policies achieves strong performance on complex, long-horizon tasks, matching or surpassing existing hierarchical TD and Monte Carlo methods.

Technology Category

• Machine Learning: Reinforcement Learning
• Reasoning under Uncertainty: Sequential Decision Making
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: User modeling for targeted and personalized online advertising

Abstract

Standard approaches to goal-conditioned reinforcement learning (GCRL) that rely on temporal-difference learning can be unstable and sample-inefficient due to bootstrapping. While recent work has explored contrastive and supervised formulations to improve stability, we present a probabilistic alternative, called survival value learning (SVL), that reframes GCRL as a survival learning problem by modeling the time-to-goal from each state as a probability distribution. This structured distributional Monte Carlo perspective yields a closed-form identity that expresses the goal-conditioned value function as a discounted sum of survival probabilities, enabling value estimation via a hazard model trained via maximum likelihood on both event and right-censored trajectories. We introduce three practical value estimators, including finite-horizon truncation and two binned infinite-horizon approximations to capture long-horizon objectives. Experiments on offline GCRL benchmarks show that SVL combined with hierarchical actors matches or surpasses strong hierarchical TD and Monte Carlo baselines, excelling on complex, long-horizon tasks.

Problem

• goal-conditioned reinforcement learning
• temporal-difference learning
• sample inefficiency
• instability
• bootstrapping

Innovation

• Survival Value Learning
• Goal-Conditioned Reinforcement Learning
• Distributional Monte Carlo
• Hazard Model
• Right-Censored Trajectories