ZOQO: Zero-Order Quantized Optimization

2025-01-12

AI Summary

To address the high computational and memory overhead of deep learning training in resource-constrained settings, this paper proposes Zeroth-Order Quantized Optimization (ZO-QO): a training framework that eliminates full-precision gradient computation entirely, instead leveraging only zeroth-order gradient sign approximations for optimization—while maintaining end-to-end low-bit quantization of both parameters and activations. ZO-QO is the first method to jointly design zeroth-order optimization with dual quantization of parameters and activations, thereby removing reliance on full-precision floating-point arithmetic throughout training. It integrates four key components: zeroth-order gradient estimation, quantization-aware constrained optimization, sign-based parameter updates, and low-bit co-training. Evaluated on large language model fine-tuning and black-box adversarial attack tasks, ZO-QO matches full-precision baseline performance while reducing GPU memory consumption by 3.2× and computational cost by 2.8×, significantly enhancing training efficiency and hardware compatibility.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Optimization
• Computer Vision: Learning & Optimization for CV

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The increasing computational and memory demands in deep learning present significant challenges, especially in resource-constrained environments. We introduce a zero-order quantized optimization (ZOQO) method designed for training models with quantized parameters and operations. Our approach leverages zero-order approximations of the gradient sign and adapts the learning process to maintain the parameters' quantization without the need for full-precision gradient calculations. We demonstrate the effectiveness of ZOQO through experiments in fine-tuning of large language models and black-box adversarial attacks. Despite the limitations of zero-order and quantized operations training, our method achieves competitive performance compared to full-precision methods, highlighting its potential for low-resource environments.

Problem

• Resource-limited environments
• Deep learning model optimization
• Precision reduction

Innovation

• ZOQO
• low-precision training
• resource-limited environments