Learning Dynamics of Zeroth-Order Optimization: A Kernel Perspective

2026-05-05

AI Summary

Zeroth-order optimization methods, despite theoretical limitations tied to parameter dimensionality, have demonstrated remarkable success in fine-tuning billion-scale language models. This work provides the first kernel-based perspective on their learning dynamics by deriving a first-order approximation of zeroth-order stochastic gradient descent, revealing that the empirical neural tangent kernel (eNTK) governs the optimization trajectory. Crucially, the approximation error depends only on the output dimension, not the parameter dimension. By integrating random subspace projection with the Johnson–Lindenstrauss lemma, the authors theoretically establish that the fidelity of zeroth-order methods is determined solely by the number of perturbations, independent of model size. This result offers a rigorous theoretical foundation for the scalability of zeroth-order optimization in large-scale models, effectively bridging the gap between theory and practice.

Abstract

Classical optimization theory establishes that zeroth-order (ZO) algorithms suffer from a dimension-dependent slowdown, with convergence rates typically scaling with the model dimension compared to first-order methods. However, in contrast to these theoretical expectations, a growing body of recent work demonstrates the successful application of ZO methods to fine-tuning Large Language Models (LLMs) with billions of parameters. To explain this paradox, we derive the one-step learning dynamics of ZO SGD, where the empirical Neural Tangent Kernel (eNTK) naturally emerges as the key term governing the learning behavior. Inspection of the eNTK produced by ZO SGD reveals that each element corresponds to the inner product of neural tangent vectors projected onto a random low-dimensional subspace. Thus, by invoking the Johnson-Lindenstrauss Lemma, our analysis shows that the fidelity of the ZO eNTK is governed primarily by the number of perturbations. Crucially, the approximation error depends on the model output size rather than the massive parameter dimension. This dimension-free property provides a theoretical justification for the scalability of ZO methods to LLMs finetuning tasks. We believe that this kernel-based framework offers a novel perspective for understanding ZO methods within the context of learning dynamics.

Problem

• zeroth-order optimization
• learning dynamics
• dimensionality curse
• large language models
• Neural Tangent Kernel

Innovation

• Zeroth-Order Optimization
• Neural Tangent Kernel
• Johnson-Lindenstrauss Lemma
• Learning Dynamics
• Large Language Models