Proximal Oracles for Optimization and Sampling

2024-04-02

arXiv.org

AI Summary

This work addresses nonsmooth convex optimization and nonsmooth log-concave sampling, focusing on Hölder-continuous (i.e., semismooth) and composite objective/potential functions. We propose a unified proximal computational framework that integrates the proximal point method with the alternating sampling framework (ASF), yielding an adaptive proximal bundle method—requiring no prior knowledge of smoothness parameters—and a novel proximal sampling oracle. For the first time, we establish non-asymptotic iteration complexity bounds for both proximal mappings and sampling oracles in the semismooth and composite settings, introducing a refined Gaussian integration technique to precisely characterize sampling error. Our results deliver universal, parameter-free optimization algorithms and the first proximal MCMC sampler with rigorous theoretical guarantees, achieving tight non-asymptotic complexity upper bounds for both problems.

Technology Category

• Search and Optimization: Non-convex Optimization
• Reasoning under Uncertainty: Stochastic Optimization
• Constraint Satisfaction and Optimization: Mixed Discrete/Continuous Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Security and Privacy: Data transparency and provenance
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

We consider convex optimization with non-smooth objective function and log-concave sampling with non-smooth potential (negative log density). In particular, we study two specific settings where the convex objective/potential function is either semi-smooth or in composite form as the finite sum of semi-smooth components. To overcome the challenges caused by non-smoothness, our algorithms employ two powerful proximal frameworks in optimization and sampling: the proximal point framework for optimization and the alternating sampling framework (ASF) that uses Gibbs sampling on an augmented distribution. A key component of both optimization and sampling algorithms is the efficient implementation of the proximal map by the regularized cutting-plane method. We establish the iteration-complexity of the proximal map in both semi-smooth and composite settings. We further propose an adaptive proximal bundle method for non-smooth optimization. The proposed method is universal since it does not need any problem parameters as input. Additionally, we develop a proximal sampling oracle that resembles the proximal map in optimization and establish its complexity using a novel technique (a modified Gaussian integral). Finally, we combine this proximal sampling oracle and ASF to obtain a Markov chain Monte Carlo method with non-asymptotic complexity bounds for sampling in semi-smooth and composite settings.

Problem

• Optimizing non-smooth convex functions with Hölder smooth properties
• Sampling from log-concave distributions with non-smooth potentials
• Developing proximal methods for both optimization and sampling challenges

Innovation

• Proximal point framework for non-smooth optimization
• Alternating sampling framework with Gibbs sampling
• Regularized cutting-plane method for proximal maps