Zeroth-Order Optimization Finds Flat Minima

2025-06-05

AI Summary

Zeroth-order optimization (ZOO) methods lack theoretical characterization of fine-grained properties of convergent solutions, particularly regarding implicit regularization mechanisms. Method: We conduct a rigorous theoretical analysis of standard two-point-estimate ZOO, leveraging Hessian trace analysis and convex/smooth optimization theory. Contribution/Results: We establish, for the first time, that ZOO implicitly prefers “flat minima”—specifically, global optima minimizing the Hessian trace—among all global minimizers. We provide formal convergence guarantees and derive explicit convergence rate bounds for such solutions. Empirical validation across convex classification tasks and large language model fine-tuning demonstrates that ZOO consistently yields flatter, better-generalizing solutions compared to gradient-based methods; moreover, the theoretically predicted convergence rates align closely with empirical observations.

Technology Category

• Machine Learning: Optimization
• Search and Optimization: Non-convex Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Zeroth-order methods are extensively used in machine learning applications where gradients are infeasible or expensive to compute, such as black-box attacks, reinforcement learning, and language model fine-tuning. Existing optimization theory focuses on convergence to an arbitrary stationary point, but less is known on the implicit regularization that provides a fine-grained characterization on which particular solutions are finally reached. We show that zeroth-order optimization with the standard two-point estimator favors solutions with small trace of Hessian, which is widely used in previous work to distinguish between sharp and flat minima. We further provide convergence rates of zeroth-order optimization to approximate flat minima for convex and sufficiently smooth functions, where flat minima are defined as the minimizers that achieve the smallest trace of Hessian among all optimal solutions. Experiments on binary classification tasks with convex losses and language model fine-tuning support our theoretical findings.

Problem

• Zeroth-order optimization favors flat minima solutions
• Convergence rates to flat minima for smooth functions
• Empirical validation on classification and language tasks

Innovation

• Zeroth-order optimization finds flat minima
• Standard two-point estimator favors small Hessian trace
• Convergence rates provided for convex smooth functions