Online Inference in Distributional Temporal-Difference Learning

2026-08-14

AI Summary

This study addresses the challenge of online statistical inference for return distribution functionals under fixed policies in reinforcement learning. Based on single-trajectory nonparametric distributed temporal difference learning, we construct a Polyak-Ruppert averaged estimator and establish its root-T convergence rate in Cramér space alongside bootstrap limit theory, thereby proving asymptotic normality. A key contribution is overcoming the bottleneck of local asymptotic inference for non-smooth functionals. Furthermore, we validate the efficacy of bootstrap methods for complex functionals such as Conditional Value-at-Risk and quantiles. Collectively, this work provides rigorous theoretical foundations and a robust online inference framework for uncertainty quantification in reinforcement learning.

Abstract

We study online statistical inference for functionals of the return distribution under a fixed policy. The return distribution is estimated by nonparametric distributional temporal-difference learning from a single Markov trajectory. For the Polyak--Ruppert averaged estimator, we prove that its root-$T$ error converges weakly to a centered Gaussian random element in Cramér space. We also prove that, conditionally on the observed trajectory, the root-$T$ difference between the bootstrap and original averages converges weakly to the same Gaussian limit. These results justify bootstrap inference for smooth statistical functionals, including variance, CVaR, expected shortfall, and expectiles. For nonsmooth statistical functionals, we develop a local asymptotic theory for the estimated return CDF over $T^{-1/2}$-neighborhoods of finitely many thresholds, together with its bootstrap analogue. This theory allows us to conduct inference for nonsmooth statistical functionals characterized by CDF equations, including return quantiles.

Problem

• Distributional Temporal-Difference Learning
• Online Statistical Inference
• Return Distribution
• Bootstrap
• Statistical Functionals

Innovation

• Distributional Temporal-Difference Learning
• Online Statistical Inference
• Bootstrap
• Local Asymptotic Theory
• Nonsmooth Functionals