Learning in Markovian bandits with non-observable states and constrained decision epochs

2026-06-25

AI Summary

This study addresses the Markovian multi-armed bandit problem under partial observability and decision-time constraints by investigating a class of self-restarting models. It establishes, for the first time in the absence of state observations, a regret bound independent of the number of states and proves the asymptotic optimality of optimal pure strategies. The authors propose UCB-NOM, an optimistic principle-based algorithm that achieves a near-logarithmic expected regret bound of ω(log T) without prior knowledge, and attains O(log T) expected regret alongside O(√(T log T)) worst-case regret when a prior bound on the bias function is available. This work demonstrates that even when strict logarithmic regret is unattainable, effective approximation remains possible, offering a novel analytical framework for Markov bandits with unobservable states.

Abstract

This paper studies the problem of regret minimization in Markovian bandits with \emph{non-observable states} and possibly \emph{constrained} decision epochs. The focus is restricted to a ``pure'' regret benchmark, that compares the performance of the learning algorithm to the best \emph{pure policy} which -- akin to optimal policies of stochastic bandits -- picks the optimal arm from start to finish without ever switching. We introduce a generalization of rested Markovian bandits, \emph{self-degrading Markovian bandits}, for which pure policies are always asymptotically optimal.We show that without prior knowledge on the underlying bandit, the regret of algorithms that switch arms rarely necessarily scales super-logarithmically for every bandit, i.e., as $ω(\log(T))$, where $T$ is the learning horizon. Despite the unreachability of the logarithmic regime, we design UCB-NOM, an optimistic algorithm inspired by UCB, of which the regret is nearly logarithmic. Lastly, we show that given prior knowledge on the Markovian bandit in the form of a bound on the bias functions of its arm, a proper instantiation of UCB-NOM achieves $O(\log(T))$ regret. We further show that this prior knowledge allows for a $O(\sqrt{T \log(T)})$ worst-case regret bound for UCB-NOM. Notably, our regret bounds do not depend on the number of states of the underlying Markov chains. Our findings suggest that the non-observability of states is a mild inconvenience in self-degrading Markovian bandits.

Problem

• Markovian bandits
• non-observable states
• constrained decision epochs
• regret minimization
• pure policy

Innovation

• non-observable states
• self-degrading Markovian bandits
• pure policy
• UCB-NOM
• regret minimization