DOPL: Direct Online Preference Learning for Restless Bandits with Preference Feedback

2024-10-07

International Conference on Learning Representations

AI Summary

In non-stationary restless multi-armed bandits (RMABs), reward functions are often difficult to model accurately due to environmental dynamics and lack of observable scalar rewards. Method: We propose Pref-RMAB, a novel framework where the decision-maker receives only pairwise preference feedback—not explicit rewards—thereby substantially reducing reliance on prior knowledge. Our approach integrates preference-based learning, confidence-interval-driven adaptive exploration, and Restless Markov state modeling. Contribution/Results: We design the first online learning algorithm for this setting with rigorous theoretical guarantees, achieving a sublinear regret bound of $ ilde{mathcal{O}}(sqrt{T ln T})$. This establishes provable convergence and near-optimal performance over time. Empirical evaluation across diverse dynamic resource allocation tasks demonstrates significant improvements over existing baselines. Pref-RMAB thus introduces a new paradigm for real-time sequential decision-making under reward-unobserved conditions.

Technology Category

• Machine Learning: Online Learning & Bandits
• Reasoning under Uncertainty: Sequential Decision Making
• Multiagent Systems: Mechanism Design

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Restless multi-armed bandits (RMAB) has been widely used to model constrained sequential decision making problems, where the state of each restless arm evolves according to a Markov chain and each state transition generates a scalar reward. However, the success of RMAB crucially relies on the availability and quality of reward signals. Unfortunately, specifying an exact reward function in practice can be challenging and even infeasible. In this paper, we introduce Pref-RMAB, a new RMAB model in the presence of preference signals, where the decision maker only observes pairwise preference feedback rather than scalar reward from the activated arms at each decision epoch. Preference feedback, however, arguably contains less information than the scalar reward, which makes Pref-RMAB seemingly more difficult. To address this challenge, we present a direct online preference learning (DOPL) algorithm for Pref-RMAB to efficiently explore the unknown environments, adaptively collect preference data in an online manner, and directly leverage the preference feedback for decision-makings. We prove that DOPL yields a sublinear regret. To our best knowledge, this is the first algorithm to ensure $ ilde{mathcal{O}}(sqrt{Tln T})$ regret for RMAB with preference feedback. Experimental results further demonstrate the effectiveness of DOPL.

Problem

• Develops Pref-RMAB model for decision-making with preference feedback
• Addresses challenge of learning from pairwise preferences instead of rewards
• Proposes DOPL algorithm to achieve sublinear regret in online learning

Innovation

• Direct online preference learning algorithm
• Adaptive online preference data collection
• Sublinear regret with preference feedback