Stochastic Linear Contextual Bandits with Bounded Noise: A Set-Membership Approach

2026-06-18

AI Summary

This work addresses the limitation of existing methods in stochastic linear contextual bandits, which typically assume sub-Gaussian reward noise despite the often stronger prior knowledge that the noise is bounded. To exploit this boundedness explicitly, the paper proposes a novel algorithm, SME-OFU, which integrates set-membership estimation (SME) to precisely quantify parameter uncertainty and combines it with the optimism in the face of uncertainty (OFU) principle for decision-making. Theoretical analysis establishes that SME-OFU achieves a logarithmic regret bound of $O(\log T)$, surpassing the performance limits inherent to sub-Gaussian assumptions. Empirical simulations further demonstrate its significant improvement over state-of-the-art algorithms designed under sub-Gaussian noise assumptions.

Abstract

This paper considers stochastic linear contextual bandits (SLCB) with bounded reward noise. Existing works typically assume sub-Gaussian reward noise and bounded expected rewards, under which the optimal regret bound scales as $\tilde{O}(\sqrt{T})$ in terms of horizon $T$. However, in many applications, realized/observed rewards are also naturally bounded, implying bounded reward noise. Bounded noise is more informative than the sub-Gaussian condition but has not been leveraged explicitly in the SLCB literature. In this paper, we propose a novel algorithm SME-OFU by utilizing an uncertainty quantification method called set-membership estimation (SME) and applying the principle of optimism in the face of uncertainty (OFU). Our algorithm enjoys an improved regret bound $O(\log T)$. Notice that this does not contradict the existing optimal bound $\tilde{O}(\sqrt{T})$ for sub-Gaussian noise because bounded noise is a stronger condition. Finally, simulations show empirical improvements of SME-OFU over a benchmark algorithm designed for sub-Gaussian noise when the reward noise is bounded.

Problem

• stochastic linear contextual bandits
• bounded noise
• reward boundedness
• regret bound
• set-membership estimation

Innovation

• set-membership estimation
• bounded noise
• linear contextual bandits
• optimism in the face of uncertainty
• regret bound