Periodic Regularized Q-Learning

2026-02-03

AI Summary

This work addresses the lack of convergence guarantees in Q-learning with linear function approximation. The authors propose Projected Regularized Q-learning (PRQ), which introduces a periodic regularization mechanism at the level of the projection operator, transforming the iterative update into a contraction mapping and thereby ensuring algorithmic stability. Building upon this framework, they develop a sampling-based reinforcement learning algorithm and, by leveraging stochastic approximation theory, establish the first finite-time convergence guarantee for Q-learning under linear approximation. The key innovation lies in the integration of regularized projection operators with contraction mapping theory, effectively resolving the long-standing challenge of convergence in stochastic environments that plagues classical approaches.

Technology Category

• Machine Learning: Reinforcement Learning
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

In reinforcement learning (RL), Q-learning is a fundamental algorithm whose convergence is guaranteed in the tabular setting. However, this convergence guarantee does not hold under linear function approximation. To overcome this limitation, a significant line of research has introduced regularization techniques to ensure stable convergence under function approximation. In this work, we propose a new algorithm, periodic regularized Q-learning (PRQ). We first introduce regularization at the level of the projection operator and explicitly construct a regularized projected value iteration (RP-VI), subsequently extending it to a sample-based RL algorithm. By appropriately regularizing the projection operator, the resulting projected value iteration becomes a contraction. By extending this regularized projection into the stochastic setting, we establish the PRQ algorithm and provide a rigorous theoretical analysis that proves finite-time convergence guarantees for PRQ under linear function approximation.

Problem

• Q-learning
• linear function approximation
• convergence
• reinforcement learning
• regularization

Innovation

• regularized projection
• contraction mapping
• linear function approximation
• finite-time convergence
• Q-learning