Active Context Selection Improves Simple Regret in Contextual Bandits

2026-05-19

AI Summary

This study addresses the contextual multi-armed bandit problem in a finite context space, where the learner can actively select contexts to minimize context-weighted simple regret. For both known and unknown context distributions, the work establishes the first tight regret bounds under active and passive sampling and proposes an optimal active sampling strategy given by $q_j \propto p_j^{2/3}$. Inspired by experimental design principles, the authors develop an Explore-Then-Commit algorithm, termed EETC, which achieves theoretically optimal performance under unknown context distributions. The analysis demonstrates that active sampling can yield up to a $\Theta(k^{1/4})$ improvement in regret compared to passive sampling. Moreover, EETC asymptotically matches the optimal regret bound attainable when the context distribution is known, and empirical results corroborate its practical advantages.

Abstract

We study the contextual multi-armed bandit problem with a finite context space (a.k.a. subpopulations), where the learner recommends a best action for each context and is evaluated by context-weighted simple regret. Our guarantees are worst-case over the reward distributions, while remaining instance-dependent with respect to the context distribution vector $p$. Akin to experimental design problems where the population of interest is fixed but the sampled subpopulation can be controlled, we allow the learner to actively choose which context to sample from. For a known $p$, we characterize tight regret rates: passive sampling where contexts are randomly revealed achieves regret of order $\sqrt{n/T \, \lVert p \rVert_{1/2}}$, whereas active sampling with allocation $q_j \propto p_j^{2/3}$ achieves the tight rate $\sqrt{n/T} \, \lVert p \rVert_{2/3}$. The resulting improvement can be as large as $Θ(k^{1/4})$, where $k$ is the number of contexts. We further extend the analysis to budgeted active sampling, characterize the corresponding tight rate, and identify when a limited active budget suffices to recover the fully active rate. When $p$ is unknown, we propose the Explore-Explore-Then-Commit (EETC) algorithm, which optimally balances estimating the context distribution and the time to switch to active allocation, such that for large horizons, it matches the known-$p$ active rate up to constants. Experiments on synthetic and real-world data support our theoretical findings.

Problem

• contextual bandits
• active sampling
• simple regret
• context distribution
• experimental design

Innovation

• active context selection
• contextual bandits
• simple regret
• optimal allocation
• instance-dependent bounds