Tight Lower Bounds under Asymmetric High-Order Hölder Smoothness and Uniform Convexity

2024-09-16

International Conference on Learning Representations

AI Summary

This work establishes tight oracle complexity lower bounds for unconstrained optimization under asymmetric smoothness and convexity assumptions: functions whose $p$-th derivative is $ u$-Hölder continuous (with parameter $H$) and which are $q$-uniformly convex (with parameter $sigma$), where $q eq p + u$. For both regimes—$q > p + u$ and $q < p + u$—we construct novel hard instances via $ell_infty$-ball-truncated Gaussian smoothing. Leveraging high-order Taylor expansions, precise characterizations of uniform convexity, and information-theoretic arguments, we derive tight lower bounds of order $Omegaig(1/varepsilon^{(p+ u)/(q-p)}ig)$. These results unify and generalize existing lower bounds for first- and higher-order smooth/convex optimization, and match the best-known upper bounds exactly. Consequently, the optimal convergence rate for this class of asymmetric optimization problems is fully characterized.

Technology Category

• Search and Optimization: Non-convex Optimization
• Constraint Satisfaction and Optimization: Constraint Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Security and Privacy: Large-scale security measurements

Abstract

In this paper, we provide tight lower bounds for the oracle complexity of minimizing high-order H""older smooth and uniformly convex functions. Specifically, for a function whose $p^{th}$-order derivatives are H""older continuous with degree $ u$ and parameter $H$, and that is uniformly convex with degree $q$ and parameter $sigma$, we focus on two asymmetric cases: (1) $q>p + u$, and (2) $q<p+ u$. Given up to $p^{th}$-order oracle access, we establish worst-case oracle complexities of $Omegaleft( left( frac{H}{sigma} ight)^frac{2}{3(p+ u)-2}left( frac{sigma}{epsilon} ight)^frac{2(q-p- u)}{q(3(p+ u)-2)} ight)$ in the first case with an $ell_infty$-ball-truncated-Gaussian smoothed hard function and $Omegaleft(left(frac{H}{sigma} ight)^frac{2}{3(p+ u)-2}+ log^2left(frac{sigma^{p+ u}}{H^q} ight)^frac{1}{p+ u-q} ight)$ in the second case, for reaching an $epsilon$-approximate solution in terms of the optimality gap. Our analysis generalizes previous lower bounds for functions under first- and second-order smoothness as well as those for uniformly convex functions, and furthermore our results match the corresponding upper bounds in the general setting.

Problem

• Tight lower bounds for minimizing high-order Hölder smooth functions
• Oracle complexity for uniformly convex functions with asymmetric cases
• Generalization of previous bounds for first- and second-order smoothness

Innovation

• Tight lower bounds for high-order Hölder smooth functions
• Asymmetric cases analysis with uniform convexity
• Generalized lower bounds matching upper bounds