Best-Arm Identification in Unimodal Bandits

2024-11-04

arXiv.org

AI Summary

This paper studies the best-arm identification problem in unimodal structured multi-armed bandits—where arm means first increase then decrease with index—under a fixed-confidence setting. Methodologically, we propose novel variants of Track-and-Stop and Top Two algorithms tailored to unimodality, integrating optimal stopping theory, large-deviations analysis, and adaptive sampling. We establish, for the first time, an instance-dependent lower bound that depends only on three critical arms, revealing the fundamental bottleneck imposed by confidence requirements. Under exponential-family and Gaussian assumptions, our algorithms achieve asymptotic and non-asymptotic optimality, respectively. Crucially, they match the derived lower bound exactly, ensuring theoretical tightness, while demonstrating superior empirical performance over standard baselines. The work thus bridges rigorous information-theoretic foundations with practical algorithmic efficiency for unimodal bandits.

Technology Category

• Machine Learning: Online Learning & Bandits
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Security and Privacy: Large-scale security measurements

Abstract

We study the fixed-confidence best-arm identification problem in unimodal bandits, in which the means of the arms increase with the index of the arm up to their maximum, then decrease. We derive two lower bounds on the stopping time of any algorithm. The instance-dependent lower bound suggests that due to the unimodal structure, only three arms contribute to the leading confidence-dependent cost. However, a worst-case lower bound shows that a linear dependence on the number of arms is unavoidable in the confidence-independent cost. We propose modifications of Track-and-Stop and a Top Two algorithm that leverage the unimodal structure. Both versions of Track-and-Stop are asymptotically optimal for one-parameter exponential families. The Top Two algorithm is asymptotically near-optimal for Gaussian distributions and we prove a non-asymptotic guarantee matching the worse-case lower bound. The algorithms can be implemented efficiently and we demonstrate their competitive empirical performance.

Problem

• Identify best-arm in unimodal bandits efficiently
• Derive lower bounds on algorithm stopping time
• Propose asymptotically optimal Top Two algorithms

Innovation

• Leverages unimodal structure for efficiency
• Modifies Track-and-Stop for optimality
• Implements Top Two algorithm near-optimally