Gap-Dependent Bounds for Q-Learning using Reference-Advantage Decomposition

2024-10-10

arXiv.org

AI Summary

This paper investigates gap-dependent regret bounds for Q-learning in finite-horizon tabular MDPs. Focusing on UCB-Advantage and Q-EarlySettled-Advantage, we propose a novel analytical framework that integrates variance-aware confidence intervals with reference–advantage decomposition. Under benign MDP structures featuring strictly positive suboptimality gaps, we establish the first $O(log T)$ gap-dependent regret upper bound—significantly improving upon the classical $sqrt{T}$ worst-case bound—and derive the first gap-dependent upper bound on policy-switching cost. Our core contributions are: (1) synergistically leveraging variance-aware confidence intervals and reference–advantage decomposition to sharpen Q-learning convergence analysis; and (2) introducing a gap-dependent error decomposition paradigm that jointly characterizes exploration efficiency and policy stability.

Technology Category

• Machine Learning: Online Learning & Bandits
• Game Theory and Economic Paradigms: Adversarial Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Social networks and social learning
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

We study the gap-dependent bounds of two important algorithms for on-policy Q-learning for finite-horizon episodic tabular Markov Decision Processes (MDPs): UCB-Advantage (Zhang et al. 2020) and Q-EarlySettled-Advantage (Li et al. 2021). UCB-Advantage and Q-EarlySettled-Advantage improve upon the results based on Hoeffding-type bonuses and achieve the almost optimal $sqrt{T}$-type regret bound in the worst-case scenario, where $T$ is the total number of steps. However, the benign structures of the MDPs such as a strictly positive suboptimality gap can significantly improve the regret. While gap-dependent regret bounds have been obtained for Q-learning with Hoeffding-type bonuses, it remains an open question to establish gap-dependent regret bounds for Q-learning using variance estimators in their bonuses and reference-advantage decomposition for variance reduction. We develop a novel error decomposition framework to prove gap-dependent regret bounds of UCB-Advantage and Q-EarlySettled-Advantage that are logarithmic in $T$ and improve upon existing ones for Q-learning algorithms. Moreover, we establish the gap-dependent bound for the policy switching cost of UCB-Advantage and improve that under the worst-case MDPs. To our knowledge, this paper presents the first gap-dependent regret analysis for Q-learning using variance estimators and reference-advantage decomposition and also provides the first gap-dependent analysis on policy switching cost for Q-learning.

Problem

• Establishes gap-dependent regret bounds for Q-learning using variance estimators.
• Develops a novel error decomposition framework for logarithmic regret bounds.
• Analyzes gap-dependent policy switching cost for Q-learning algorithms.

Innovation

• Novel error decomposition framework for Q-learning
• Gap-dependent regret bounds using variance estimators
• Reference-advantage decomposition for variance reduction