Revisiting Weighted Strategy for Non-stationary Parametric Bandits

2023-03-05

International Conference on Artificial Intelligence and Statistics

AI Summary

This work addresses the limitations of existing approaches to non-stationary parametric bandits and Markov decision processes (MDPs), where inadequate analysis of weighted policies often leads to algorithmic complexity and suboptimal efficiency. To overcome this, the paper introduces a refined analytical framework for weighted policy optimization that integrates weighted regression, function approximation, and dynamic regret analysis. This framework substantially simplifies algorithm design while achieving tighter dynamic regret bounds across multiple settings: it matches the performance of sliding-window or restart-based methods in linear bandits, improves the regret bound in generalized linear bandits (GLBs) from ~O(T^{4/5}) to ~O(T^{3/4}), and provides the first dynamic regret guarantees based on weighted policies for two classes of non-stationary MDPs.

Technology Category

• Machine Learning: Online Learning & Bandits
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

Non-stationary parametric bandits have attracted much attention recently. There are three principled ways to deal with non-stationarity, including sliding-window, weighted, and restart strategies. As many non-stationary environments exhibit gradual drifting patterns, the weighted strategy is commonly adopted in real-world applications. However, previous theoretical studies show that its analysis is more involved and the algorithms are either computationally less efficient or statistically suboptimal. This paper revisits the weighted strategy for non-stationary parametric bandits. In linear bandits (LB), we discover that this undesirable feature is due to an inadequate regret analysis, which results in an overly complex algorithm design. We propose a refined analysis framework, which simplifies the derivation and importantly produces a simpler weight-based algorithm that is as efficient as window/restart-based algorithms while retaining the same regret as previous studies. Furthermore, our new framework can be used to improve regret bounds of other parametric bandits, including Generalized Linear Bandits (GLB) and Self-Concordant Bandits (SCB). For example, we develop a simple weighted GLB algorithm with an $\widetilde{O}(k_\mu^{\frac{5}{4}} c_\mu^{-\frac{3}{4}} d^{\frac{3}{4}} P_T^{\frac{1}{4}}T^{\frac{3}{4}})$ regret, improving the $\widetilde{O}(k_\mu^{2} c_\mu^{-1}d^{\frac{9}{10}} P_T^{\frac{1}{5}}T^{\frac{4}{5}})$ bound in prior work, where $k_\mu$ and $c_\mu$ characterize the reward model's nonlinearity, $P_T$ measures the non-stationarity, $d$ and $T$ denote the dimension and time horizon.

Problem

• non-stationary bandits
• weighted strategy
• parametric bandits
• Markov Decision Processes
• regret analysis

Innovation

• weighted strategy
• non-stationary bandits
• refined analysis framework
• dynamic regret
• function approximation in MDPs