Concave Statistical Utility Maximization Bandits via Influence-Function Gradients

2026-04-23

AI Summary

This work addresses the limitation of classical multi-armed bandits, which optimize only expected rewards and thus fail to capture uncertainty in the long-term reward distribution. The authors propose a novel framework that directly optimizes concave distributional utilities—such as variance and Wasserstein distance—by reformulating the infinite-horizon problem as a mixed-strategy optimization over the probability simplex. The key innovation lies in the first-time application of influence function calculus to construct unbiased gradient estimators, effectively decoupling optimization error from estimation bias. An entropy mirror-ascent algorithm is then employed for efficient policy updates. Theoretical analysis yields regret bounds, while experiments demonstrate the method’s efficacy under both variance and Wasserstein objectives, further comparing the performance of exact versus plug-in influence functions.

Technology Category

• Machine Learning: Online Learning & Bandits
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Learning to Search

Application Category

• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

We study stochastic multi-armed bandits in which the objective is a statistical functional of the long-run reward distribution, rather than expected reward alone. Under mild continuity assumptions, we show that the infinite-horizon problem reduces to optimizing over stationary mixed policies: each weight vector \(w\) on the simplex induces a mixture law \(P^w\), and performance is measured by the concave utility \(U(w)=\mathfrak U(P^w)\). For differentiable statistical utilities, we use influence-function calculus to derive stochastic gradient estimators from bandit feedback. This leads to an entropic mirror-ascent algorithm on a truncated simplex, implemented through multiplicative-weights updates and plug-in estimates of the influence function. We establish regret bounds that separate the mirror-ascent optimization error from the bias caused by estimating the influence function. The framework is developed for general concave distributional utilities and illustrated through variance and Wasserstein objectives, with numerical experiments comparing exact and plug-in influence-function implementations.

Problem

• statistical utility
• multi-armed bandits
• concave optimization
• distributional objectives
• stochastic bandits

Innovation

• influence-function gradients
• concave statistical utility
• mirror-ascent bandits
• distributional optimization
• stochastic multi-armed bandits