On the Variance of Temporal Difference Learning and its Reduction Using Control Variates

2026-06-18

AI Summary

This study investigates the high-variance issue in temporal-difference (TD) learning within reinforcement learning and its impact on estimation stability. Under tabular representations and episodic settings, the authors theoretically derive and numerically validate, for the first time, an asymptotic upper bound on the variance of TD estimates relative to Monte Carlo (MC) methods, demonstrating that shorter update horizons effectively reduce variance. They further interpret Direct Advantage Estimation (DAE) as a regression-adjusted control variate and establish a tighter asymptotic variance bound than TD under large-sample conditions. The findings indicate that aggregating multiple trajectories, reducing update horizons, and incorporating DAE all substantially improve variance control, with empirical validation of estimator variance behavior conducted in tailored environments.

Abstract

We analyze the variance of temporal difference (TD) learning using the phased setting with tabular representation, and show that one of the mechanisms behind its ability to reduce variance is by effectively aggregating over a larger number of independent trajectories. Based on this insight, we demonstrate that (1) the variance of TD is asymptotically bounded from above by Monte Carlo (MC) estimators, and (2) shorter horizon updates incurs less variance for a fixed number of samples. Beyond TD, we show that Direct Advantage Estimation (DAE), a method for estimating the advantage function, can be seen as a type of regression-adjusted control variate, which achieves a tighter bound on the variance compared to TD in the large-sample limit. Finally, we numerically illustrate the behaviors of these estimators with carefully designed environments.

Problem

• temporal difference learning
• variance reduction
• control variates
• Monte Carlo estimators
• advantage function estimation

Innovation

• Temporal Difference Learning
• Variance Reduction
• Control Variates
• Direct Advantage Estimation
• Monte Carlo Estimators