Residuals-based Offline Reinforcement Learning

2026-04-01

AI Summary

This work addresses the instability of policies in offline reinforcement learning caused by insufficient data coverage and distributional shift. The authors propose a residual-based offline reinforcement learning framework that explicitly models dynamic estimation error to construct an empirical residual, which is then used to define a contractive residual Bellman operator. This operator is theoretically guaranteed to possess an asymptotically optimal fixed point and finite-sample convergence. By integrating empirical residual estimation, a reformulated Bellman equation, and deep Q-networks, the framework enables stable policy learning without any environment interaction. Experimental results on a stochastic CartPole environment demonstrate that the proposed residual offline DQN algorithm significantly outperforms existing methods.

Abstract

Offline reinforcement learning (RL) has received increasing attention for learning policies from previously collected data without interaction with the real environment, which is particularly important in high-stakes applications. While a growing body of work has developed offline RL algorithms, these methods often rely on restrictive assumptions about data coverage and suffer from distribution shift. In this paper, we propose a residuals-based offline RL framework for general state and action spaces. Specifically, we define a residuals-based Bellman optimality operator that explicitly incorporates estimation error in learning transition dynamics into policy optimization by leveraging empirical residuals. We show that this Bellman operator is a contraction mapping and identify conditions under which its fixed point is asymptotically optimal and possesses finite-sample guarantees. We further develop a residuals-based offline deep Q-learning (DQN) algorithm. Using a stochastic CartPole environment, we demonstrate the effectiveness of our residuals-based offline DQN algorithm.

Problem

• offline reinforcement learning
• distribution shift
• data coverage
• Bellman operator
• policy optimization

Innovation

• residuals-based
• offline reinforcement learning
• Bellman optimality operator
• distribution shift
• deep Q-learning