Accelerated Stochastic Min-Max Optimization Based on Bias-corrected Momentum

2024-06-18

arXiv.org

AI Summary

This work addresses nonconvex-strongly-concave (or nonconvex-PL) stochastic minimax optimization, aiming to break the existing $O(varepsilon^{-4})$ stochastic first-order oracle complexity barrier. We propose the first bias-corrected momentum framework tailored to this setting, which integrates efficient Hessian-vector product estimation with stochastic first-order updates—requiring neither second-order derivatives nor additional memory overhead. Theoretically, we establish an improved iteration complexity of $O(varepsilon^{-3})$, achieving the first acceleration result for this class of problems. Empirical evaluation on robust logistic regression confirms substantial improvements in convergence rate, with strong alignment between theoretical guarantees and practical performance.

Technology Category

• Search and Optimization: Non-convex Optimization
• Reasoning under Uncertainty: Stochastic Optimization
• Machine Learning: Optimization

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Lower-bound analyses for nonconvex strongly-concave minimax optimization problems have shown that stochastic first-order algorithms require at least $mathcal{O}(varepsilon^{-4})$ oracle complexity to find an $varepsilon$-stationary point. Some works indicate that this complexity can be improved to $mathcal{O}(varepsilon^{-3})$ when the loss gradient is Lipschitz continuous. The question of achieving enhanced convergence rates under distinct conditions, remains unresolved. In this work, we address this question for optimization problems that are nonconvex in the minimization variable and strongly concave or Polyak-Lojasiewicz (PL) in the maximization variable. We introduce novel bias-corrected momentum algorithms utilizing efficient Hessian-vector products. We establish convergence conditions and demonstrate a lower iteration complexity of $mathcal{O}(varepsilon^{-3})$ for the proposed algorithms. The effectiveness of the method is validated through applications to robust logistic regression using real-world datasets.

Problem

• Improving convergence rates for nonconvex-strongly-concave minimax optimization
• Reducing oracle complexity to O(ε⁻³) using bias-corrected momentum
• Validating method via robust logistic regression on real-world datasets

Innovation

• Bias-corrected momentum for min-max optimization
• Utilizes efficient Hessian-vector products
• Achieves lower iteration complexity O(ε⁻³)