Model Predictive Control is Almost Optimal for Restless Bandit

2024-10-08

arXiv.org

AI Summary

This paper studies the discrete-time infinite-horizon average-reward Restless Markovian Bandit (RMAB) problem. We propose the first model predictive control (MPC)-based receding-horizon optimization policy: at each time step, it solves a τ-step finite-horizon linear program and executes only the first action. Innovatively introducing MPC to RMAB, we establish a novel theoretical analysis framework grounded in dissipativity. Under general conditions, we prove a suboptimality gap of $O(1/sqrt{N})$; under local stability, the gap improves to exponential convergence $exp(-Omega(N))$. The policy is conceptually simple, computationally efficient, and empirically outperforms existing methods by a significant margin. Moreover, it extends naturally to general constrained MDP settings.

Technology Category

• Machine Learning: Online Learning & Bandits
• Reasoning under Uncertainty: Stochastic Optimization
• Planning, Routing, and Scheduling: Planning with Markov Models (MDPs, POMDPs)

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

We consider the discrete time infinite horizon average reward restless markovian bandit (RMAB) problem. We propose a emph{model predictive control} based non-stationary policy with a rolling computational horizon $ au$. At each time-slot, this policy solves a $ au$ horizon linear program whose first control value is kept as a control for the RMAB. Our solution requires minimal assumptions and quantifies the loss in optimality in terms of $ au$ and the number of arms, $N$. We show that its sub-optimality gap is $O(1/sqrt{N})$ in general, and $exp(-Omega(N))$ under a local-stability condition. Our proof is based on a framework from dynamic control known as emph{dissipativity}. Our solution easy to implement and performs very well in practice when compared to the state of the art. Further, both our solution and our proof methodology can easily be generalized to more general constrained MDP settings and should thus, be of great interest to the burgeoning RMAB community.

Problem

• Develops a model predictive control policy for restless bandit problems
• Quantifies sub-optimality gap in terms of horizon and arm count
• Generalizes solution to constrained Markov decision processes

Innovation

• Model Predictive Control for RMAB
• Rolling horizon linear programming
• Dissipativity-based proof framework