🤖 AI Summary
Q-learning with linear function approximation often diverges under off-policy training and bootstrapping (e.g., the Baird counterexample). This paper proposes a novel Q-learning algorithm incorporating an ℓ₂ regularization term and provides the first rigorous proof that appropriate regularization ensures global convergence in the linear approximation setting. Methodologically, the algorithm’s update dynamics are modeled as a switched linear system, and convergence is established via Lyapunov stability analysis, yielding explicit sufficient conditions for convergence and a closed-form bound on the approximation error. Theoretically, this work establishes ℓ₂ regularization as a sufficient condition for guaranteeing convergence of linear Q-learning—resolving a long-standing issue in reinforcement learning theory. Empirically, the proposed algorithm demonstrates stable convergence in canonical divergent settings (e.g., Baird’s counterexample), significantly outperforming standard linear Q-learning in both stability and asymptotic accuracy.
📝 Abstract
Q-learning is widely used algorithm in reinforcement learning community. Under the lookup table setting, its convergence is well established. However, its behavior is known to be unstable with the linear function approximation case. This paper develops a new Q-learning algorithm that converges when linear function approximation is used. We prove that simply adding an appropriate regularization term ensures convergence of the algorithm. We prove its stability using a recent analysis tool based on switching system models. Moreover, we experimentally show that it converges in environments where Q-learning with linear function approximation has known to diverge. We also provide an error bound on the solution where the algorithm converges.