Efficient Tuning Before Low-Bit Post-Training Quantization for Stochastic Gradient Descent-optimized Models

2026-07-13

AI Summary

This work addresses the significant performance degradation often observed in low-bit (e.g., 2/4-bit) post-training quantization (PTQ), which stems from the sensitivity of full-precision models to quantization error. To mitigate this issue, the authors propose Efficient Tuning Before Quantization (ETBQ), a lightweight pre-tuning method that injects simulated quantization noise during full-precision training. This guides optimization toward loss landscapes that are inherently robust to quantization, thereby enhancing subsequent PTQ performance. Notably, ETBQ avoids explicit fake quantization operations and yields full-precision models compatible with any PTQ backend without modification. Experimental results demonstrate consistent improvements across benchmarks: under W2A4 settings, it achieves a 2.14% gain in top-1 accuracy on Tiny-ImageNet and a 5.80% increase in mIoU on Cityscapes, validating its effectiveness and broad applicability.

Abstract

Post-training quantization (PTQ) compresses deep neural networks for deployment under limited memory and computational budgets. However, low-bit (i.e., 2-bit or 4-bit) PTQ often suffers from substantial performance degradation. Most existing PTQ methods operate on an unconstrained full-precision (FP) model and primarily address quantization errors through post-hoc reconstruction. We argue that low-bit PTQ accuracy is limited not only by post-quantization error minimization, but also by the quantization-error tolerance of a FP model itself. In this paper, we propose Efficient Tuning Before Quantization (ETBQ), a pre-conditioning tuning stage for Stochastic Gradient Descent (SGD)-optimized models before PTQ. During tuning, the FP model is optimized under perturbations sampled from the error distributions of weight and activation quantization, guiding the model toward a loss-landscape region that is less sensitive to the subsequent PTQ. Unlike QAT, ETBQ does not train a fake-quantized deployment model, which is computationally and memory intensive. Instead, ETBQ outputs a FP model that can be used by any PTQ backend. Experiments on CIFAR-100, Tiny-ImageNet, ImageNet, and Cityscapes provide consistent evidence that ETBQ improves low-bit PTQ across diverse tasks. Under W2A4 settings, e.g., ETBQ improves over naive PTQ by 2.14\% top-1 accuracy on Tiny-ImageNet and by 5.80\% mIoU on Cityscapes. Code is available at https://github.com/xpxpxp2001xpxpxp/ETBQ.

Problem

• post-training quantization
• low-bit quantization
• quantization-error tolerance
• SGD-optimized models
• performance degradation

Innovation

• post-training quantization
• low-bit quantization
• quantization-aware tuning
• SGD-optimized models
• error tolerance