Self-Normalized Inference for Constant-Stepsize Temporal-Difference Learning under Markovian Sampling

2026-08-11

AI Summary

This work addresses the inferential challenges in fixed-stepsize linear temporal difference (TD) learning under Markovian sampling—specifically, the complications arising from serial dependence and stepsize-dependent stationary targets. By establishing a functional central limit theorem and integrating Richardson–Romberg extrapolation with Brownian bridge self-normalization, the authors construct a single-pass, covariance-free asymptotically pivotal confidence region. The proposed method eliminates the need for long-run covariance estimation, avoids tuning parameters such as bandwidth or batch length, and admits a constant-memory online implementation. Through a cross-iteration stepsize design, it enables valid inference on the projected Bellman solution. Empirical evaluations in FrozenLake and Garnet environments demonstrate accurate coverage, effective Richardson–Romberg bias correction, and strong finite-sample performance.

Abstract

Constant-stepsize temporal-difference (TD) learning is attractive for policy evaluation, but inference from a single Markov trajectory must account for serial dependence and a stepsize-dependent stationary target. For fixed-stepsize linear TD, we establish a functional central limit theorem whose covariance retains the multiplicative component induced by the random TD matrix and the stationary iterate error. We then derive a joint functional limit for parallel Richardson--Romberg (RR) recursions driven by the same trajectory. A Brownian-bridge self-normalizer yields asymptotically pivotal confidence regions for prespecified state-value contrasts without estimating the long-run covariance or selecting a bandwidth or batch length. For such a contrast, the procedure admits a one-pass implementation whose memory does not grow with the trajectory length. At a fixed stepsize, the inferential center is the RR stationary target. We also study horizon-indexed designs in which the stepsize remains constant within each run and decreases across longer horizons. Under an explicit RR-dependent rate window, the residual RR target shift, multiplicative remainder, and initialization effect are negligible at the root-$n$ scale, yielding inference for the projected Bellman solution. Experiments on FrozenLake and Garnet illustrate stationary-target coverage, RR target correction, and the finite-sample behavior of the horizon-indexed design.

Problem

• temporal-difference learning
• self-normalized inference
• Markovian sampling
• stationary target
• policy evaluation

Innovation

• Temporal-Difference Learning
• Self-Normalization
• Richardson–Romberg Extrapolation
• Functional Central Limit Theorem
• Markovian Sampling