How Log-Barrier Helps Exploration in Policy Optimization

2026-03-16

AI Summary

This work addresses a critical limitation in existing stochastic gradient bandit (SGB) algorithms, whose convergence to the global optimum relies on the unrealistic assumption that the optimal action is selected with a probability bounded away from zero. To overcome this, we propose the Log-Barrier Stochastic Gradient Bandit (LB-SGB) method, which incorporates a log-barrier regularizer into the policy optimization objective to structurally enforce a minimal level of exploration. LB-SGB guarantees convergence without requiring a positive lower bound on the optimal action’s selection probability, while maintaining sample complexity comparable to that of standard SGB. Theoretical analysis reveals an intrinsic connection between the log-barrier regularization and natural policy gradients through their shared exploitation of the Fisher information geometry of the policy space. Empirical results demonstrate that LB-SGB achieves significantly improved exploration efficiency and convergence robustness.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Machine Learning: Online Learning & Bandits
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

Recently, it has been shown that the Stochastic Gradient Bandit (SGB) algorithm converges to a globally optimal policy with a constant learning rate. However, these guarantees rely on unrealistic assumptions about the learning process, namely that the probability of the optimal action is always bounded away from zero. We attribute this to the lack of an explicit exploration mechanism in SGB. To address these limitations, we propose to regularize the SGB objective with a log-barrier on the parametric policy, structurally enforcing a minimal amount of exploration. We prove that Log-Barrier Stochastic Gradient Bandit (LB-SGB) matches the sample complexity of SGB, but also converges (at a slower rate) without any assumptions on the learning process. We also show a connection between the log-barrier regularization and Natural Policy Gradient, as both exploit the geometry of the policy space by controlling the Fisher information. We validate our theoretical findings through numerical simulations, showing the benefits of the log-barrier regularization.

Problem

• exploration
• policy optimization
• stochastic gradient bandit
• log-barrier
• convergence assumptions

Innovation

• log-barrier regularization
• policy optimization
• exploration
• stochastic gradient bandit
• natural policy gradient