A Diffusion Approximation for Temporal-Difference Learning with Linear Features under Markovian Noise

2026-06-16

AI Summary

This work addresses the steady-state error of linear TD(0) under Markovian noise, which arises from sample dependence and cannot be adequately captured by traditional ordinary differential equation models that overlook the impact of stochastic fluctuations on the error floor. The paper introduces, for the first time, a diffusion approximation to construct a stochastic differential equation (SDE) model that accurately characterizes the dynamics of constant-stepsize TD(0). This framework cleanly separates the contractive effect of the projected Bellman operator from the random perturbations induced by Markovian noise. The analysis reveals that the lower bound of the steady-state error is jointly determined by the long-term covariance structure of the Markov process and the geometric properties of the projected Bellman operator, thereby offering a novel theoretical foundation for understanding the convergence and stability of TD learning in settings with dependent samples.

Abstract

Temporal difference (TD) learning with linear function approximation is a core method for policy evaluation. Its classical continuous-time description is an ordinary differential equation (ODE), which captures the asymptotic mean dynamics but neglects stochastic fluctuations determining the error floor. We introduce a stochastic differential equation (SDE) approximation for linear TD(0) under Markovian noise. The resulting model distinguishes the contraction dynamics governed by the projected Bellman operator from the influence of Markovian sampling. As a consequence, the model explains the constant-stepsize error floor through the interaction between Markovian long-run covariance and the contraction geometry of the projected Bellman operator.

Problem

• Temporal-difference learning
• Linear function approximation
• Markovian noise
• Error floor
• Stochastic fluctuations

Innovation

• stochastic differential equation
• temporal-difference learning
• Markovian noise
• error floor
• projected Bellman operator