GINO-Q: Learning an Asymptotically Optimal Index Policy for Restless Multi-armed Bandits

2024-08-19

arXiv.org

AI Summary

Traditional reinforcement learning fails in large-scale restless multi-armed bandits (RMABs) due to exponential explosion of state and action spaces under non-stationarity. Method: We propose the first three-timescale stochastic approximation algorithm for RMABs that does not require the indexability assumption. Our approach leverages single-arm decomposition and generalizes Whittle index learning, enabling linear scalability in the number of arms. Contribution/Results: The algorithm breaks the classical Whittle index’s strong structural dependency, achieving near-global optimality even in non-indexable settings. We establish its asymptotic optimality via rigorous convergence analysis. Empirical evaluation demonstrates several-fold acceleration in convergence speed and supports deployment at scale—up to ten thousand arms—making it the first index-based solution for large-scale dynamic resource allocation that simultaneously provides theoretical guarantees and practical scalability.

Technology Category

• Machine Learning: Online Learning & Bandits
• Reasoning under Uncertainty: Stochastic Optimization
• Multiagent Systems: Mechanism Design

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

The restless multi-armed bandit (RMAB) framework is a popular model with applications across a wide variety of fields. However, its solution is hindered by the exponentially growing state space (with respect to the number of arms) and the combinatorial action space, making traditional reinforcement learning methods infeasible for large-scale instances. In this paper, we propose GINO-Q, a three-timescale stochastic approximation algorithm designed to learn an asymptotically optimal index policy for RMABs. GINO-Q mitigates the curse of dimensionality by decomposing the RMAB into a series of subproblems, each with the same dimension as a single arm, ensuring that complexity increases linearly with the number of arms. Unlike recently developed Whittle-index-based algorithms, GINO-Q does not require RMABs to be indexable, enhancing its flexibility and applicability. Our experimental results demonstrate that GINO-Q consistently learns near-optimal policies, even for non-indexable RMABs where Whittle-index-based algorithms perform poorly, and it converges significantly faster than existing baselines.

Problem

• Overcoming exponential state space growth in RMABs
• Solving combinatorial action space in large-scale RMABs
• Learning optimal policies for non-indexable RMABs

Innovation

• Three-timescale stochastic approximation algorithm
• Decomposes RMAB into single-arm subproblems
• Works without requiring indexability condition