🤖 AI Summary
Existing convergence analyses of Q-learning and its smooth variants (e.g., soft Q-learning) rely heavily on restrictive assumptions such as quasimonotonicity and struggle to uniformly handle asynchronous settings.
Method: This paper establishes a unified stability analysis framework based on ordinary differential equations (ODEs), abandoning conventional switched-system approaches that require quasimonotonicity. Instead, it introduces a $p$-norm Lyapunov function and integrates asynchronous stochastic iteration modeling with generalized stability theory.
Contributions/Results: First, it provides the first unified asymptotic convergence proof for both standard Q-learning and multiple entropy-regularized variants. Second, it rigorously establishes convergence for asynchronous Q-learning and its smooth counterparts under significantly milder conditions. Third, it delivers a scalable, non-monotonic, and asynchronous theoretical analysis paradigm applicable to a broad class of entropy-regularized reinforcement learning algorithms.
📝 Abstract
Convergence of Q-learning has been the focus of extensive research over the past several decades. Recently, an asymptotic convergence analysis for Q-learning was introduced using a switching system framework. This approach applies the so-called ordinary differential equation (ODE) approach to prove the convergence of the asynchronous Q-learning modeled as a continuous-time switching system, where notions from switching system theory are used to prove its asymptotic stability without using explicit Lyapunov arguments. However, to prove stability, restrictive conditions, such as quasi-monotonicity, must be satisfied for the underlying switching systems, which makes it hard to easily generalize the analysis method to other reinforcement learning algorithms, such as the smooth Q-learning variants. In this paper, we present a more general and unified convergence analysis that improves upon the switching system approach and can analyze Q-learning and its smooth variants. The proposed analysis is motivated by previous work on the convergence of synchronous Q-learning based on $p$-norm serving as a Lyapunov function. However, the proposed analysis addresses more general ODE models that can cover both asynchronous Q-learning and its smooth versions with simpler frameworks.