High-dimensional Linear Bandits with Knapsacks

2023-11-02

International Conference on Machine Learning

AI Summary

This paper studies the high-dimensional sparse contextual bandits with knapsack constraints (CBwK), a sequential decision-making problem under resource (knapsack) capacity constraints, where both rewards and resource consumption exhibit high-dimensional (d-dimensional) linear structures with inherent sparsity. We propose the first algorithm that integrates online hard-thresholding sparse estimation with a primal-dual framework: dual variables are updated online while simultaneously applying hard-thresholding projections to adaptively exploit sparsity. Theoretically, the algorithm achieves a cumulative regret of $ ilde{O}(sqrt{T} + log d)$, improving the dimension dependence from the conventional polynomial rate to a logarithmic one—$O(log d)$—and attaining the optimal $ ilde{O}(sqrt{T})$ rate in the unconstrained special case. Empirical evaluations confirm its efficiency and robustness in high-dimensional sparse settings.

Abstract

We study the contextual bandits with knapsack (CBwK) problem under the high-dimensional setting where the dimension of the feature is large. The reward of pulling each arm equals the multiplication of a sparse high-dimensional weight vector and the feature of the current arrival, with additional random noise. In this paper, we investigate how to exploit this sparsity structure to achieve improved regret for the CBwK problem. To this end, we first develop an online variant of the hard thresholding algorithm that performs the sparse estimation in an online manner. We further combine our online estimator with a primal-dual framework, where we assign a dual variable to each knapsack constraint and utilize an online learning algorithm to update the dual variable, thereby controlling the consumption of the knapsack capacity. We show that this integrated approach allows us to achieve a sublinear regret that depends logarithmically on the feature dimension, thus improving the polynomial dependency established in the previous literature. We also apply our framework to the high-dimension contextual bandit problem without the knapsack constraint and achieve optimal regret in both the data-poor regime and the data-rich regime. We finally conduct numerical experiments to show the efficient empirical performance of our algorithms under the high dimensional setting.

Problem

• High-dimensional linear bandits with knapsack constraints
• Sparse estimation for sharper regret guarantees
• Dual-variable scheme for budget-aware resource consumption

Innovation

• Online sparse estimation via hard thresholding
• Primal-dual scheme for budget constraints
• Dual resolving for sharper regret bounds