Unraveling Zeroth-Order Optimization through the Lens of Low-Dimensional Structured Perturbations

2025-01-31

AI Summary

To address the slow convergence of zeroth-order optimization in high-dimensional and large-model tuning—caused by high-variance gradient estimates—this paper proposes a unified optimization framework based on structured perturbations (sparsity, low-rankness, and block coordinate descent). It establishes, for the first time, a joint theoretical analysis of convergence and generalization for zeroth-order methods under structured perturbations. Two novel geometric metrics—“stable rank” and “effective overlap”—are introduced to characterize the intrinsic roles of structure in noise reduction and acceleration. Theoretically, block coordinate descent (BCD) is proven to be the optimal structured strategy, and the first generalization error upper bound for such methods is derived. The proposed algorithm, MeZO-BCD, maintains memory efficiency while achieving a 2.09× speedup in wall-clock and iteration time, with accuracy matching or exceeding prior approaches.

Technology Category

• Machine Learning: Optimization
• Search and Optimization: Non-convex Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience

Abstract

Zeroth-order (ZO) optimization has emerged as a promising alternative to gradient-based backpropagation methods, particularly for black-box optimization and large language model (LLM) fine-tuning. However, ZO methods suffer from slow convergence due to high-variance stochastic gradient estimators. While structured perturbations, such as sparsity and low-rank constraints, have been explored to mitigate these issues, their effectiveness remains highly under-explored. In this work, we develop a unified theoretical framework that analyzes both the convergence and generalization properties of ZO optimization under structured perturbations. We show that high dimensionality is the primary bottleneck and introduce the notions of extit{stable rank} and extit{effective overlap} to explain how structured perturbations reduce gradient noise and accelerate convergence. Using the uniform stability under our framework, we then provide the first theoretical justification for why these perturbations enhance generalization. Additionally, through empirical analysis, we identify that extbf{block coordinate descent} (BCD) to be an effective structured perturbation method. Extensive experiments show that, compared to existing alternatives, memory-efficient ZO (MeZO) with BCD ( extit{MeZO-BCD}) can provide improved converge with a faster wall-clock time/iteration by up to $ imes extbf{2.09}$ while yielding similar or better accuracy.

Problem

• Zeroth-order Optimization
• High-dimensional Space
• Sparse and Low-rank Constraints

Innovation

• Zeroth-order Optimization
• Block Coordinate Descent
• Memory-efficient Optimization (MeZO-BCD)