Horizon Adaptive Offline Policy Learning via Value Stitching

2026-06-19

AI Summary

This work addresses the bias and inflexibility in value estimation inherent to traditional temporal difference learning when applied to long-horizon tasks, stemming from fixed bootstrapping horizons. To overcome this limitation, the authors propose a dynamic-horizon adaptive off-policy learning method that recursively composes return segments of varying lengths and leverages a conditional auxiliary value function. By anchoring on actionable subgoal states, the approach dynamically selects both the optimal horizon length and subgoals to construct more accurate value supervision signals. Empirical evaluation demonstrates that the proposed method significantly outperforms baseline approaches—including fixed-horizon, extended-horizon, and generative-value offline reinforcement learning—across 50 tasks in the OGBench benchmark, with particularly pronounced gains in high-complexity, long-horizon scenarios.

Abstract

Learning accurate value functions plays a decisive role for reinforcement learning (RL) agents to solve long-horizon, complex tasks. Conventional temporal-difference (TD) learning objectives suffer from value-estimation bias that accumulates over the horizon, while extended-horizon modeling methods, such as n-step TD backups and Q-chunking, adopt a rigid, fixed-horizon value-modeling recipe that is often not flexible enough to capture complex value structures in long-horizon, multi-stage tasks. In this paper, we show that enabling value updates with dynamic horizon composition can yield a strong offline policy learning scheme. Our method, Horizon Adaptive Offline Policy Learning via VAlue STitching (VAST), replaces fixed-horizon backups with recursive, horizon-adaptive value composition. Its key ingredient is to couple value optimization with a future state- and horizon-length-conditioned auxiliary value function that is learned through direct data supervision, and a stitching policy that optimally selects the reward-maximizing horizon length and future sub-goal to achieve horizon-adaptive value stitching. This design enables direct estimation and compositional "stitching" of variable-length returns grounded in actionable sub-goal states, providing an accurate and greedily exploitable value-supervision signal for offline policy optimization. Across 50 tasks on OGBench, VAST outperforms fixed-step, extended-horizon methods, and generative-value offline RL baselines, achieving strong performance particularly in high-complexity, long-horizon decision-making tasks.

Problem

• offline reinforcement learning
• long-horizon tasks
• value estimation bias
• fixed-horizon modeling
• multi-stage decision-making

Innovation

• horizon-adaptive
• value stitching
• offline reinforcement learning
• dynamic horizon composition
• auxiliary value function