🤖 AI Summary
This paper identifies a fundamental limitation of TD(λ): although it minimizes mean-squared error in state-value estimates, policy quality depends critically on the *relative ordering* of state values—not their absolute accuracy. Consequently, even when initialized with the optimal policy, TD(λ) may converge to suboptimal policies, as empirically demonstrated in two- and three-state MDPs and Backgammon. To address this, we propose a novel framework that directly optimizes *state-value ranking consistency*, introducing STD(λ)—the first algorithm to formulate temporal-difference learning as a pairwise ranking problem. STD(λ) constructs a differentiable ranking loss from binary comparisons and integrates principles from both TD learning and contrastive (differential) training. We prove that, in two-state systems, STD(λ) guarantees monotonic policy improvement and convergence to the optimal policy. Experiments confirm that STD(λ) significantly outperforms TD(λ) in both synthetic two-state MDPs and a modified Acrobot task, and further reveal that TD(λ)’s value misranking is the root cause of its policy degradation in Backgammon.
📝 Abstract
TD( ) with function approximation has proved empirically successful for some complex reinforcement learning problems. For linear approximation, TD( ) has been shown to minimise the squared error between the approximate value of each state and the true value. However, as far as policy is concerned, it is error in the relative ordering of states that is critical, rather than error in the state values. We illustrate this point, both in simple two-state and three-state systems in which TD( )—starting from an optimal policy—converges to a sub-optimal policy, and also in backgammon. We then present a modified form of TD( ), called STD( ), in which function approximators are trained with respect to relative state values on binary decision problems. A theoretical analysis, including a proof of monotonic policy improvement for STD( ) in the context of the two-state system, is presented, along with a comparison with Bertsekas’ differential training method [1]. This is followed by successful demonstrations of STD( ) on the two-state system and a variation on the well known acrobot problem.