Bridging the Gap Between Average and Discounted TD Learning

2026-05-03

AI Summary

In average-reward reinforcement learning, temporal difference (TD) methods face convergence challenges due to the absence of a contractive Bellman operator. This work proposes a novel algorithm based on double-sample trajectories that constructs a convergent projected Bellman equation, enabling—for the first time—a unified convergence analysis aligned with discounted TD under the average-reward setting. The method eliminates dependence on the state-space dimensionality, reducing sample complexity from quartic to quadratic. It guarantees convergence to a unique solution in both tabular and linear function approximation settings, achieving sample efficiency comparable to discounted TD, with complexity scaling only quadratically in the condition number.

Abstract

The analysis of Temporal Difference (TD) learning in the average-reward setting faces notable theoretical difficulties because the Bellman operator is not contractive with respect to any norm. This complicates standard analyses of stochastic updates that are effective in discounted settings. Although a considerable body of literature addresses these challenges, existing theoretical approaches come with limitations. We introduce a novel algorithm designed explicitly for policy evaluation in the average-reward setting, utilizing sampling from two Markovian trajectories. Our proposed method overcomes previous limitations by guaranteeing convergence to the unique solution of a properly defined projected Bellman equation. Notably, and in contrast to earlier work, our convergence analysis is uniformly applicable to both linear function approximation and tabular settings and does not involve explicit dimension-dependent terms in its convergence bounds. These results align with what is known to hold in the discounted setting. Furthermore, our algorithm achieves improved dependence on the problem's condition number, reducing the sample complexity from quartic, as in prior literature, to quadratic scaling, and thus matching the efficiency seen in the discounted setting.

Problem

• average-reward
• Temporal Difference learning
• Bellman operator
• non-contractive
• policy evaluation

Innovation

• average-reward TD learning
• projected Bellman equation
• sample complexity
• condition number
• linear function approximation