Algorithm Design and Stronger Guarantees for the Improving Multi-Armed Bandits Problem

2025-11-13

AI Summary

This paper studies the multi-armed bandit problem with monotonically increasing rewards exhibiting diminishing marginal returns (i.e., concave reward functions), aiming to improve resource allocation efficiency under uncertainty. To overcome the limitation of existing algorithms—which only guarantee Ω(k) or Ω(√k) approximation ratios in the worst case—we propose two parameterized algorithm families that achieve, for the first time, *k*-dependent optimal data-dependent guarantees. Under mild concavity conditions, our algorithms break classical approximation lower bounds, attaining O(1) or O(log k) approximation ratios on benign instances. Our approach unifies stochastic and deterministic strategies, integrating offline data-driven learning with statistical learning principles to characterize the interplay between reward curve properties and sample complexity. The framework is broadly applicable to sequential decision-making domains including technology R&D, clinical trial design, and hyperparameter optimization.

Technology Category

• Machine Learning: Online Learning & Bandits
• Reasoning under Uncertainty: Stochastic Optimization
• Game Theory and Economic Paradigms: Mechanism Design

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

The improving multi-armed bandits problem is a formal model for allocating effort under uncertainty, motivated by scenarios such as investing research effort into new technologies, performing clinical trials, and hyperparameter selection from learning curves. Each pull of an arm provides reward that increases monotonically with diminishing returns. A growing line of work has designed algorithms for improving bandits, albeit with somewhat pessimistic worst-case guarantees. Indeed, strong lower bounds of $Omega(k)$ and $Omega(sqrt{k})$ multiplicative approximation factors are known for both deterministic and randomized algorithms (respectively) relative to the optimal arm, where $k$ is the number of bandit arms. In this work, we propose two new parameterized families of bandit algorithms and bound the sample complexity of learning the near-optimal algorithm from each family using offline data. The first family we define includes the optimal randomized algorithm from prior work. We show that an appropriately chosen algorithm from this family can achieve stronger guarantees, with optimal dependence on $k$, when the arm reward curves satisfy additional properties related to the strength of concavity. Our second family contains algorithms that both guarantee best-arm identification on well-behaved instances and revert to worst case guarantees on poorly-behaved instances. Taking a statistical learning perspective on the bandit rewards optimization problem, we achieve stronger data-dependent guarantees without the need for actually verifying whether the assumptions are satisfied.

Problem

• Designing algorithms for improving multi-armed bandits with stronger guarantees
• Addressing pessimistic worst-case bounds in bandit optimization problems
• Achieving data-dependent guarantees without verifying assumption satisfaction

Innovation

• Parameterized algorithm families with sample complexity bounds
• Optimal dependence on k for concave reward curves
• Best-arm identification with worst-case fallback guarantees