Policy Optimization Achieves Data-Dependent Regret Bounds in MDPs with Unknown Transitions

2026-06-30

AI Summary

This work addresses the challenge of simultaneously adapting to both adversarial and stochastic losses in online tabular Markov decision processes with unknown transition kernels, aiming to achieve data-dependent regret bounds. To this end, the authors propose a novel algorithm based on optimistic Follow-the-Regularized-Leader (FTRL), which, for the first time in the unknown-transition setting, attains data-dependent regret guarantees through policy optimization. The method integrates optimistic Q-function estimation, data-dependent transition-reward terms, and a bias-control mechanism, introducing a transition-related complexity measure. Theoretical analysis shows that the algorithm simultaneously achieves first-order, second-order, and path-length regret bounds incorporating this complexity term, and further obtains a polylogarithmic-in-horizon regret bound dependent on the suboptimality gap under stochastic environments.

Abstract

We study policy optimization for online episodic tabular Markov decision processes with unknown transition kernels, aiming for best-of-both-worlds guarantees together with data-dependent regret bounds. Recent work (Dann et al., 2023; Li et al., 2026) has shown that policy optimization can adapt to both adversarial and stochastic losses with first-order, second-order, and path-length bounds, but only under known transitions, leaving open whether such data-dependent guarantees are achievable by policy optimization when the transition kernel is unknown. We resolve this by developing a new algorithm based on optimistic follow-the-regularized-leader that attains these guarantees under unknown transitions. The key ingredient is a new design of optimistic $Q$-function estimators together with a data-dependent transition bonus that controls estimator bias through the loss-prediction error. Our analysis further identifies an unavoidable transition-dependent complexity term that captures the intrinsic cost of estimating the transition kernel. As a result, we obtain first-order, second-order, and path-length bounds with the transition-dependent complexity term while simultaneously achieving gap-dependent $\mathrm{polylog}(T)$ regret in the stochastic regime.

Problem

• policy optimization
• unknown transitions
• data-dependent regret bounds
• Markov decision processes
• best-of-both-worlds

Innovation

• policy optimization
• data-dependent regret
• unknown transitions
• optimistic Q-function
• transition-dependent complexity