Lyapunov-Certified Direct Switching Theory for Q-Learning

2026-04-21

AI Summary

This work addresses the lack of tight and computable theoretical guarantees in existing finite-time convergence analyses of Q-learning. By modeling constant-stepsize Q-learning for the first time as a stochastic switched linear system driven by martingale difference noise, the authors exploit the equivalence between Bellman error and stochastic policy dynamics to construct a Lyapunov analysis framework based on the joint spectral radius. This approach yields a tight finite-time bound on the final iteration error, where the convergence rate is precisely characterized by the joint spectral radius—strictly improving upon conventional row-sum bounds—and provides a verifiable quadratic Lyapunov certificate for stability and performance.

Technology Category

• Machine Learning: Online Learning & Bandits
• Multiagent Systems: Multiagent Learning
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Large-scale security measurements

Abstract

Q-learning is one of the most fundamental algorithms in reinforcement learning. We analyze constant-stepsize Q-learning through a direct stochastic switching system representation. The key observation is that the Bellman maximization error can be represented exactly by a stochastic policy. Therefore, the Q-learning error admits a switched linear conditional-mean recursion with martingale-difference noise. The intrinsic drift rate is the joint spectral radius (JSR) of the direct switching family, which can be strictly smaller than the standard row-sum rate. Using this representation, we derive a finite-time final-iterate bound via a JSR-induced Lyapunov function and then give a computable quadratic-certificate version.

Problem

• Q-learning
• Lyapunov function
• switched system
• joint spectral radius
• finite-time bound

Innovation

• Lyapunov certification
• Q-learning
• switched linear system
• joint spectral radius
• finite-time bound