Zero-order Parameter-free Optimization for LMO-based Methods: Novel Approach for Efficient Fine-tuning

2026-06-12

AI Summary

This work addresses the high memory overhead of backpropagation in large language model fine-tuning and the sensitivity of existing zeroth-order optimization methods to step size and smoothing parameters, which necessitates cumbersome hyperparameter tuning. To overcome these limitations, the paper proposes an efficient gradient-free fine-tuning approach that eliminates the need for pre-specified hyperparameters. It introduces, for the first time, a parameter-adaptive mechanism into a zeroth-order optimization framework based on the Linear Minimization Oracle (LMO), enabling non-Euclidean geometry-aware updates across heterogeneous parameter blocks. The method is validated on the OPT-1.3B model, demonstrating strong empirical performance, theoretical convergence guarantees, and substantial reductions in both memory consumption and hyperparameter tuning effort.

Abstract

Fine-tuning large language models (LLMs) has become a central application of modern optimization, enabling pretrained models to adapt to diverse downstream tasks and domain-specific data. A major obstacle in large-scale fine-tuning is the memory overhead of backpropagation, which requires storing activations, gradients, and optimizer states. Zeroth-order (ZO) optimization offers a memory-efficient alternative, but its performance is highly sensitive to the stepsize and smoothing parameter, often requiring costly task-specific tuning. Parameter-free (PF) optimization addresses this issue by adapting algorithmic parameters without prior knowledge of problem-dependent constants. Moreover, large-scale fine-tuning can benefit from geometry-aware updates that account for the heterogeneous structure of parameter blocks, which can be modeled through methods that exploit linear minimization oracle (LMO). In this work, we study PF adaptation for LMO-based ZO optimization and introduce $\texttt{AdaNAGED}$, a method that unifies gradient-free training, adaptive tuning, and non-Euclidean update geometry. We establish convergence guarantees and validate the method on large-scale LLM fine-tuning task with $\texttt{OPT}-1.3\mathrm{B}$ model.

Problem

• zeroth-order optimization
• parameter-free optimization
• large language model fine-tuning
• linear minimization oracle
• memory-efficient optimization

Innovation

• zeroth-order optimization
• parameter-free
• linear minimization oracle
• non-Euclidean geometry
• large language model fine-tuning