Sparse Nonparametric Contextual Bandits

2025-03-20

AI Summary

This paper studies the sparse nonparametric contextual bandit problem: given an infinite candidate feature set, only a small unknown subset is relevant, and the goal is to simultaneously identify these critical features and minimize cumulative regret. We first establish the minimax regret lower bound for this setting. Then, we propose an enhanced Feel-Good Thompson Sampling algorithm whose regret upper bound matches the lower bound up to logarithmic factors, achieving optimal regret control. Our approach integrates nonparametric function modeling, sparse feature selection, kernel methods, and generalization analysis of neural networks; the derived regret upper bound scales polynomially in the number of actions and logarithmically in the number of effective candidate features. Experiments demonstrate that incorporating sparsity significantly improves the long-term performance of both kernel-based and neural-network-based contextual bandits.

Technology Category

• Machine Learning: Online Learning & Bandits
• Search and Optimization: Non-convex Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience

Abstract

This paper studies the problem of simultaneously learning relevant features and minimising regret in contextual bandit problems. We introduce and analyse a new class of contextual bandit problems, called sparse nonparametric contextual bandits, in which the expected reward function lies in the linear span of a small unknown set of features that belongs to a known infinite set of candidate features. We consider two notions of sparsity, for which the set of candidate features is either countable or uncountable. Our contribution is two-fold. First, we provide lower bounds on the minimax regret, which show that polynomial dependence on the number of actions is generally unavoidable in this setting. Second, we show that a variant of the Feel-Good Thompson Sampling algorithm enjoys regret bounds that match our lower bounds up to logarithmic factors of the horizon, and have logarithmic dependence on the effective number of candidate features. When we apply our results to kernelised and neural contextual bandits, we find that sparsity always enables better regret bounds, as long as the horizon is large enough relative to the sparsity and the number of actions.

Problem

• Learning relevant features in contextual bandits
• Minimizing regret with sparse nonparametric models
• Analyzing regret bounds for kernel and neural bandits

Innovation

• Sparse nonparametric contextual bandits introduced
• Feel-Good Thompson Sampling variant analyzed
• Regret bounds improved with sparsity conditions