🤖 AI Summary
This paper studies the restless multi-armed bandit (RMAB) problem under a general observation model, capturing error-prone feedback arising from resource constraints or noise. The system dynamics are governed by a generic probabilistic observation model, and the problem is formulated as a Markov decision process over a countable belief state space. Methodologically, we extend the partial conservation laws (PCL) implementability region approach—previously restricted to finite belief spaces—to infinite belief spaces for the first time, rigorously establishing Whittle index existence, PCL-indexability, and its analytical structure. We further propose a finite-dimensional approximation algorithm with provable convergence guarantees, integrated within the Niño-Mora–Bertsimas adaptive greedy (AG) framework. Numerical experiments demonstrate that the proposed method significantly outperforms baseline policies, achieving both theoretical rigor and practical efficacy.
📝 Abstract
In this paper, we consider a general observation model for restless multi-armed bandit problems. The operation of the player needs to be based on certain feedback mechanism that is error-prone due to resource constraints or environmental or intrinsic noises. By establishing a general probabilistic model for dynamics of feedback/observation, we formulate the problem as a restless bandit with a countable belief state space starting from an arbitrary initial belief (a priori information). We apply the achievable region method with partial conservation law (PCL) to the infinite-state problem and analyze its indexability and priority index (Whittle index). Finally, we propose an approximation process to transform the problem into which the AG algorithm of Ni~no-Mora and Bertsimas for finite-state problems can be applied to. Numerical experiments show that our algorithm has an excellent performance.
