Bilevel ZOFO: Bridging Parameter-Efficient and Zeroth-Order Techniques for Efficient LLM Fine-Tuning and Meta-Training

2025-02-05

AI Summary

To address challenges in efficient fine-tuning of large language models (LLMs)—including the limited performance of parameter-efficient fine-tuning (PEFT), the reliance of zeroth-order (ZO) methods on handcrafted prompts, and the high gradient computation overhead of first-order (FO) meta-training—this paper proposes a novel bilevel zeroth-order–first-order (ZOFO) optimization framework. ZOFO innovatively integrates ZO gradient estimation (requiring only forward passes) with PEFT within a dual-loop architecture and provides the first theoretical convergence guarantee for ZO-based LLM fine-tuning. In single-task fine-tuning, ZOFO significantly outperforms both pure PEFT and pure ZO baselines while maintaining memory overhead comparable to PEFT. In multi-task meta-training, ZOFO achieves substantially lower computational cost than existing FO-based meta-algorithms, with comparable or superior performance.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Fine-tuning pre-trained Large Language Models (LLMs) for downstream tasks using First-Order (FO) optimizers presents significant computational challenges. Parameter-Efficient Fine-Tuning(PEFT) methods have been proposed to address these challenges by freezing most model parameters and training only a small subset. While PEFT is efficient, it may not outperform full fine-tuning when high task-specific performance is required. Zeroth-Order (ZO) methods offer an alternative for fine-tuning the entire pre-trained model by approximating gradients using only the forward pass, thus eliminating the computational burden of back-propagation in first-order methods. However, when implementing ZO methods, a hard prompt is crucial, and relying on simple, fixed hard prompts may not be optimal. In this paper, we propose a bilevel optimization framework that complements ZO methods with PEFT to mitigate sensitivity to hard prompts while efficiently and effectively fine-tuning LLMs. Our Bilevel ZOFO (Zeroth-Order-First-Order) method employs a double-loop optimization strategy, where only the gradient of the PEFT model and the forward pass of the base model are required. We provide convergence guarantees for Bilevel ZOFO. Empirically, we demonstrate that Bilevel ZOFO outperforms both PEFT and ZO methods in single-task settings while maintaining similar memory efficiency. Additionally, we show its strong potential for multitask learning. Compared to current first-order meta-training algorithms for multitask learning, our method has significantly lower computational demands while maintaining or improving performance.

Problem

• Efficiently fine-tune Large Language Models.
• Mitigate sensitivity to hard prompts in ZO methods.
• Lower computational demands in multitask learning.

Innovation

• Bilevel ZOFO optimizes LLM fine-tuning.
• Combines Zeroth-Order and PEFT techniques.
• Reduces computational demands in meta-training.