Towards Parameter-Free Temporal Difference Learning

2026-03-02

AI Summary

This work addresses the gap between theory and practice in temporal difference (TD) learning, where traditional theoretical analyses rely on hard-to-estimate problem-specific parameters—such as the minimum eigenvalue of the feature covariance matrix or the mixing time of the Markov chain. To overcome this limitation, the paper proposes a parameter-free TD(0) algorithm with exponentially decaying step sizes that requires no projection, iterate averaging, or prior knowledge of the environment. The method is proven to converge under both i.i.d. and Markov sampling settings without depending on any instance-specific constants. Notably, it achieves convergence rates comparable to existing approaches while guaranteeing an optimal bias-variance trade-off for the final iterate in the i.i.d. setting, thereby significantly narrowing the divide between theoretical guarantees and practical performance.

Technology Category

• Machine Learning: Learning with Manifolds
• Search and Optimization: Learning to Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Economics, Online Markets and Human Computation: Social networks and social learning

Abstract

Temporal difference (TD) learning is a fundamental algorithm for estimating value functions in reinforcement learning. Recent finite-time analyses of TD with linear function approximation quantify its theoretical convergence rate. However, they often require setting the algorithm parameters using problem-dependent quantities that are difficult to estimate in practice -- such as the minimum eigenvalue of the feature covariance (\(ω\)) or the mixing time of the underlying Markov chain (\(τ_{\text{mix}}\)). In addition, some analyses rely on nonstandard and impractical modifications, exacerbating the gap between theory and practice. To address these limitations, we use an exponential step-size schedule with the standard TD(0) algorithm. We analyze the resulting method under two sampling regimes: independent and identically distributed (i.i.d.) sampling from the stationary distribution, and the more practical Markovian sampling along a single trajectory. In the i.i.d.\ setting, the proposed algorithm does not require knowledge of problem-dependent quantities such as \(ω\), and attains the optimal bias-variance trade-off for the last iterate. In the Markovian setting, we propose a regularized TD(0) algorithm with an exponential step-size schedule. The resulting algorithm achieves a comparable convergence rate to prior works, without requiring projections, iterate averaging, or knowledge of \(τ_{\text{mix}}\) or \(ω\).

Problem

• Temporal Difference Learning
• Parameter-Free
• Linear Function Approximation
• Markovian Sampling
• Convergence Analysis

Innovation

• parameter-free
• temporal difference learning
• exponential step-size
• Markovian sampling
• linear function approximation