Learning under Quantization for High-Dimensional Linear Regression

2025-10-20

AI Summary

This work systematically investigates the theoretical impact of multi-type low-bit quantization—of data, labels, parameters, activations, and gradients—on learning performance in high-dimensional linear regression, addressing a critical gap in the theoretical analysis of quantized training. We propose the first unified finite-step SGD framework that jointly models quantization-induced noise amplification, spectral distortion, and approximation error. To mitigate spectral distortion, we design a multiplicative quantization scheme that provably eliminates it, and we uncover, for the first time, the batch-size scaling effect inherent to additive quantization. Our analysis yields a precise excess risk upper bound, which quantitatively demonstrates that multiplicative quantization strictly outperforms additive quantization under polynomially decaying spectra. These results establish a rigorous theoretical foundation for efficient training under hardware constraints and introduce a novel design paradigm for quantization-aware optimization.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Learning to Search
• Computer Vision: Learning & Optimization for CV

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

The use of low-bit quantization has emerged as an indispensable technique for enabling the efficient training of large-scale models. Despite its widespread empirical success, a rigorous theoretical understanding of its impact on learning performance remains notably absent, even in the simplest linear regression setting. We present the first systematic theoretical study of this fundamental question, analyzing finite-step stochastic gradient descent (SGD) for high-dimensional linear regression under a comprehensive range of quantization targets: data, labels, parameters, activations, and gradients. Our novel analytical framework establishes precise algorithm-dependent and data-dependent excess risk bounds that characterize how different quantization affects learning: parameter, activation, and gradient quantization amplify noise during training; data quantization distorts the data spectrum; and data and label quantization introduce additional approximation and quantized error. Crucially, we prove that for multiplicative quantization (with input-dependent quantization step), this spectral distortion can be eliminated, and for additive quantization (with constant quantization step), a beneficial scaling effect with batch size emerges. Furthermore, for common polynomial-decay data spectra, we quantitatively compare the risks of multiplicative and additive quantization, drawing a parallel to the comparison between FP and integer quantization methods. Our theory provides a powerful lens to characterize how quantization shapes the learning dynamics of optimization algorithms, paving the way to further explore learning theory under practical hardware constraints.

Problem

• Analyzing quantization effects on high-dimensional linear regression learning performance
• Establishing risk bounds for various quantization targets in SGD optimization
• Comparing multiplicative vs additive quantization risks for spectral distortion

Innovation

• Analyzes SGD under comprehensive quantization targets
• Establishes risk bounds for different quantization types
• Proves spectral distortion elimination in multiplicative quantization