Leveraging Similarities in Multi-Armed Bandits

2026-06-22

AI Summary

This work addresses the challenge of leveraging tree-structured action similarities—encoded such that the loss function satisfies tree compatibility—in multi-armed bandit problems, where traditional single-point feedback fails to effectively exploit this structure to reduce regret. The authors propose a unified adaptive online learning algorithm that accommodates a spectrum of multi-point feedback mechanisms, ranging from semi-bandit to minimal two-point feedback. By introducing a similarity-aware effective number of actions, denoted $K_{\text{eff}}$, to replace the original action count $K$, the algorithm achieves improved regret bounds. Theoretical analysis reveals a fundamental limitation of single-point feedback in harnessing tree similarity and establishes, for the first time, an optimal $\sqrt{T}$ regret bound for Lipschitz bandits with dimension $d \leq 2$ under two-point feedback, striking an optimal balance between generality and structural exploitation.

Abstract

In many online learning and bandit problems, the actions we consider possess inherent similarities--for instance because they share latent traits, tags, or hierarchical structure. We study online learning with a similarity-structured action set, encoded by a rooted tree whose leaves are the actions and whose levels quantify how closely two actions are related. The loss sequence is assumed tree-compatible: losses of similar actions are constrained to be close. We establish an impossibility result showing that usual one-point bandit feedback cannot, in general, leverage range or tree-induced similarity, even under very strong similarity constraints. We then provide a unified set of algorithms which adapt to a wide range of richer feedback models, from semi-bandit feedback down to multi-point bandit protocols, including the minimal two-point feedback setting. We show these algorithms exhibit best-of-both-worlds guarantees and provably exploit action similarities by replacing the number of actions $K$ by a similarity-aware effective number of actions $K_{\mathrm{eff}}$ in the regret bounds. As an application, we show that under two-point feedback, it is possible to achieve $\sqrt{T}$ regret in Lipschitz bandits when $d \leq 2$.

Problem

• multi-armed bandits
• similarity structure
• tree-compatible losses
• bandit feedback
• online learning

Innovation

• similarity-structured bandits
• tree-compatible losses
• two-point feedback
• effective number of actions
• best-of-both-worlds regret