ZO-Act: Efficient Zeroth-Order Fine-Tuning via One-Shot Activation-Informed Low-Rank Subspaces

2026-07-01

AI Summary

Existing zeroth-order (ZO) optimization methods for fine-tuning face high gradient estimation variance and performance bottlenecks when backpropagation is unavailable or memory is constrained, primarily due to perturbing full model weights or random low-dimensional subspaces. This work proposes ZO-Act, which constructs a fixed low-rank basis per linear layer during initialization based on input activations and optimizes only lightweight subspace coefficient matrices. This approach substantially reduces the perturbation dimensionality and gradient estimator variance while supporting momentum-based optimizers and quantized model fine-tuning, effectively controlling subspace approximation bias. Experiments demonstrate that ZO-Act consistently outperforms current ZO fine-tuning baselines across language understanding, question answering, and commonsense reasoning tasks on Llama-3-8B, OPT-13B, and their INT4 quantized variants.

Abstract

Zeroth-order (ZO) optimization enables fine-tuning large language models when backpropagation is unavailable or memory-prohibitive, but existing methods often perturb full model weights or randomly constructed low-dimensional subspaces, yielding high-variance estimates and limited performance. We propose ZO-Act, an activation-informed ZO fine-tuning method that restricts perturbations to a fixed low-rank subspace derived from input activations. For each linear layer, ZO-Act computes a small activation basis once at initialization and optimizes only lightweight coefficient matrices using forward-only loss evaluations. This reduces the effective perturbation dimension, exposes explicit trainable variables compatible with momentum-based optimizers such as Adam, and naturally supports quantized LLM fine-tuning by keeping low-bit weights frozen. We analyze ZO-Act as zeroth-order optimization over a restricted coefficient space and show that perturbing the low-dimensional coefficients reduces both the variance-dependent convergence term and the finite-difference error of the ZO estimator, at the cost of a controlled subspace approximation bias that is mitigated by the low-rank structure of LLM activations and gradients. Experiments on Llama-3-8B, OPT-13B, and INT4 Llama-3-8B show consistent gains over strong ZO fine-tuning baselines across language understanding, question answering, and commonsense reasoning.

Problem

• zeroth-order optimization
• large language models
• fine-tuning
• high-variance estimation
• low-dimensional subspaces

Innovation

• zeroth-order optimization
• activation-informed subspace
• low-rank adaptation
• quantized LLM fine-tuning
• forward-only optimization