Sign-Separated Finite-Time Error Analysis of Q-Learning

2026-05-15

AI Summary

Existing Q-learning theory inadequately characterizes error dynamics over finite time horizons, particularly lacking insight into how positive and negative error components distinctly influence convergence. This work proposes a sign-separation approach that decomposes the error into positive and negative parts, modeling them respectively as a linear time-invariant system and a switched system. This decomposition reveals an inherent asymmetry induced by the Bellman optimality operator: negative errors converge rapidly due to a lower bound imposed by the optimal policy, whereas positive errors are prone to amplification, leading to overestimation. Leveraging comparison system theory and constant-stepsize recursive analysis, we establish tight finite-time exponential convergence bounds, demonstrating that the convergence rate of negative errors is at least as fast as—and often faster than—that of positive errors.

Abstract

This paper develops a sign-separated finite-time error analysis for constant step-size Q-learning. Starting from the switching-system representation, the error is decomposed into its componentwise negative and positive parts. The negative part is dominated by a lower comparison linear time-invariant (LTI) system associated with a fixed optimal policy, whereas the positive part is controlled by a linear switching system. The resulting bounds show that the negative-side LTI certificate is no slower than the positive-side switching certificate and may produce a faster exponential envelope. The analysis identifies a max-induced asymmetry in Q-learning error dynamics. This asymmetry is connected to overestimation: positive action-wise errors can be selected and propagated by the Bellman maximum, whereas negative errors admit an optimal-policy lower comparison. Finite-time bounds are provided for both deterministic and stochastic constant-step-size recursions.

Problem

• Q-learning
• finite-time error analysis
• sign-separated error
• overestimation
• asymmetry

Innovation

• sign-separated analysis
• finite-time error bound
• Q-learning
• switching system
• overestimation bias