Heavy-Ball Q-Learning with Residual Weighting Correction

2026-06-25

AI Summary

This work addresses the slow convergence of standard Q-learning by proposing a heavy-ball Q-learning algorithm with residual-weighted correction, which incorporates a momentum mechanism to accelerate convergence and extends naturally to the setting of linear function approximation. Innovatively adopting the switched linear system (SLS) perspective for analyzing Q-learning dynamics, the study leverages joint spectral radius (JSR) theory to rigorously derive sufficient conditions under which the proposed algorithm achieves accelerated convergence. The theoretical analysis demonstrates that, under these conditions, the method not only guarantees convergence but also exhibits a faster convergence rate than standard Q-learning, while preserving both convergence and acceleration advantages in the linear function approximation regime.

Abstract

This paper proposes a corrected heavy-ball Q-learning method for reinforcement learning (RL) and establishes its convergence. It also identifies conditions under which the method is theoretically guaranteed to converge faster than standard Q-learning. The same construction is then extended to Q-learning with linear function approximation, where analogous convergence and acceleration statements are derived. The analysis is based on a switched linear system (SLS) representation of Q-learning algorithms and on the joint spectral radius (JSR) of the associated switching families. This SLS viewpoint is not commonly used in standard analyses of Q-learning, and it provides a complementary framework and new insight into how heavy-ball momentum can accelerate Q-learning.

Problem

• Heavy-Ball Q-Learning
• convergence
• acceleration
• reinforcement learning
• switched linear system

Innovation

• Heavy-Ball Momentum
• Switched Linear System
• Joint Spectral Radius
• Q-Learning Acceleration
• Residual Weighting Correction