From Cursed to Competitive: Closing the ZO-FO Gap via Input-to-State Stability

2026-04-28

AI Summary

This work addresses the long-standing perception that zeroth-order (ZO) optimization algorithms inherently converge more slowly than first-order (FO) methods due to dimensionality dependence. By interpreting ZO algorithms through the lens of dynamical systems, the paper models them as perturbed averaged versions of FO algorithms and introduces, for the first time, the input-to-state stability (ISS) theoretical framework to analyze their convergence behavior. Under suitable conditions, the authors rigorously establish that ZO methods can eliminate the dimensionality gap in expectation, converging at the same rate as FO methods to an arbitrarily small neighborhood of the FO fixed point. These theoretical findings are corroborated by numerical experiments, highlighting the underappreciated convergence potential of ZO optimization algorithms.

Abstract

While it is generally understood that zeroth-order (ZO) algorithms have an extra dependency on their number of iterations for any choice of parameters, compared to their first-order (FO) counterparts, in this work, we show that under several conditions, in expectation, ZO methods do not suffer from extra dimension dependencies in their convergence rates with respect to their FO counterparts. We look at optimisation algorithms from the dynamical systems perspective and analyse the conditions under which one can formulate the average of a ZO algorithm as the average of its FO counterpart with bounded perturbations with values dependent on design parameters. Then, using input-to-state stability properties, we show ZO methods follow the same decay rate as their FO counterparts and converge to a neighbourhood of the fixed point of FO methods, where its radius depends on the bound of the norm of the perturbations, which can be made arbitrarily small. The theoretical findings are illustrated via numerical examples.

Problem

• zeroth-order optimization
• first-order optimization
• convergence rate
• dimension dependence
• input-to-state stability

Innovation

• zeroth-order optimization
• input-to-state stability
• convergence rate
• dimension independence
• dynamical systems perspective