ConMeZO: Adaptive Descent-Direction Sampling for Gradient-Free Finetuning of Large Language Models

2025-11-04

AI Summary

To address slow convergence in zeroth-order fine-tuning of large language models (LLMs) caused by high-dimensional random parameter-space search, this paper proposes ConeZO: a conical momentum-guided adaptive directional sampling method. ConeZO constrains zeroth-order random direction sampling to a cone centered around a momentum-estimated direction, thereby focusing exploration on directions more likely aligned with the true gradient and mitigating the curse of dimensionality. The method requires only two forward passes per iteration and no backward propagation, with memory overhead comparable to MeZO. We provide theoretical analysis showing that ConeZO achieves the same worst-case convergence rate as MeZO. Empirical evaluation on natural language understanding and generation tasks demonstrates up to a 2× improvement in convergence speed over MeZO and superior performance relative to existing zeroth-order optimizers.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Zeroth-order or derivative-free optimization (MeZO) is an attractive strategy for finetuning large language models (LLMs) because it eliminates the memory overhead of backpropagation. However, it converges slowly due to the inherent curse of dimensionality when searching for descent directions in the high-dimensional parameter space of billion-scale LLMs. We propose ConMeZO, a novel zeroth-order optimizer that accelerates convergence by adaptive directional sampling. Instead of drawing the direction uniformly at random, ConMeZO restricts the sampling to a cone centered around a momentum estimate. This concentrates the search in directions where the true gradient is more likely to lie and thus reduces the effect of high dimensions. We prove that ConMeZO achieves the same worst-case convergence rate as MeZO. Empirically, when finetuning LLMs on natural language tasks, ConMeZO is up to 2X faster than MeZO while retaining the low-memory footprint of zeroth-order methods.

Problem

• Accelerates gradient-free finetuning of large language models
• Reduces slow convergence in high-dimensional parameter spaces
• Maintains low memory usage while improving optimization speed

Innovation

• Adaptive directional sampling for zeroth-order optimization
• Cone-based sampling around momentum estimate
• Faster convergence with low-memory footprint