model quantization

This work addresses the limitation of conventional quantization-aware training (QAT), which supports only a single numerical format and thus struggles to accommodate dynamic precision requirements during inference. To overcome this, the authors propose a multi-format QAT framework integrated with a Slice-and-Scale transformation mechanism, enabling a single model to achieve high performance across diverse MXINT and MXFP formats while allowing runtime precision switching without retraining. Anchored on high-precision formats such as MXINT8 or MXFP8, the method generalizes instantly to unseen lower-precision configurations. Experimental results demonstrate that the proposed approach matches the accuracy of dedicated single-format QAT models across all target precisions, with negligible degradation during format transitions, thereby substantially enhancing deployment flexibility.

This work addresses the challenge of deploying deep neural networks on 6G edge devices under extreme compression constraints while preserving model accuracy. Existing mixed-precision quantization methods suffer from coarse granularity, making them ill-suited to capture neuron-level variations in precision requirements. To overcome this limitation, we propose Neuron-level Mixed-Precision Quantization-Aware Training (NMP-QAT), which, for the first time, adaptively assigns discrete bit-widths to individual neurons during training—increasing precision only when necessary—and applies uniformly to both weights and activations. By integrating differentiable proxy functions, straight-through estimators, and a fully discrete inference graph, NMP-QAT significantly outperforms current mixed-precision QAT approaches on MLP and tabular models, achieving superior compression-accuracy trade-offs across both telecom and non-telecom datasets, thereby enabling greener edge AI deployment.

This work addresses the computational resource allocation between full-precision (FP) and quantized training phases in quantization-aware training (QAT), where optimal scheduling remains poorly understood. Method: We propose a tokens-per-parameter-byte loss scaling law—the first to characterize how the optimal QAT-to-FP ratio increases with total compute budget. We further introduce “QAT cooling,” a novel learning-rate-scheduling strategy that dynamically reduces redundant quantized training iterations. Combining scaling-law modeling with empirical analysis, we accurately predict optimal QAT duration and final accuracy across diverse model sizes and bit-widths. Results: Under fixed compute budgets, our method significantly improves quantized model accuracy. It also systematically uncovers the quantitative trade-off between memory footprint (governed by bit-width and QAT duration) and accuracy—enabling principled, resource-aware QAT deployment.

To address the limitations of fixed scaling factors in quantization-aware training (QAT)—including reduced inference flexibility, poor adaptability to diverse activation distributions, and accuracy degradation—this paper proposes Adaptive Scalable Quantization (ASQ). ASQ introduces three key innovations: (1) a dynamic, trainable scaling module that adapts in real time to layer-wise activation distributions; (2) a POST non-uniform quantization scheme with base $2^{1/2}$, balancing accurate modeling of bell-shaped weight distributions and hardware efficiency; and (3) lookup-table (LUT)-based inference acceleration. Evaluated on 4-bit ResNet-34/ImageNet, ASQ achieves only a 1.2% top-1 accuracy drop relative to the full-precision baseline—outperforming state-of-the-art quantization methods by a significant margin, and even surpassing the baseline in certain configurations. Moreover, ASQ enhances deployment flexibility and operational efficiency without compromising accuracy or hardware compatibility.

To address deployment constraints on resource-limited devices, this paper proposes a fine-tuning-free post-training quantization (PTQ) method for non-uniform weight quantization. To overcome the accuracy bottlenecks of conventional uniform or channel-wise quantization, we introduce, for the first time in PTQ, a theoretically grounded noise-minimization mechanism that jointly optimizes clipping thresholds and scaling factors. Our approach leverages gradient-free statistical modeling, hierarchical non-uniform clustering, and convex optimization to achieve distribution-adaptive quantization. Evaluated on real-world datasets, our method achieves 4–8× model compression, 3.2× inference speedup, and less than 0.3% Top-1 accuracy degradation—substantially outperforming state-of-the-art PTQ baselines. Key contributions include: (i) a theoretically guaranteed optimal non-uniform quantization design; (ii) the first provably noise-minimizing PTQ framework; and (iii) a highly efficient, entirely fine-tuning-free implementation.

This work proposes a method for training genuine 4-bit convolutional neural networks from scratch on standard CPUs without relying on specialized hardware, custom kernels, or post-training quantization. By integrating tanh-based soft weight clipping, symmetric quantization, dynamic per-layer scaling, and straight-through estimators—all implemented using native PyTorch operations—the approach maintains only 15 unique weight values per layer. It achieves 92.34% and 70.94% accuracy on CIFAR-10 and CIFAR-100, respectively, with less than 0.16% degradation compared to full-precision models, while delivering an 8× memory compression. Notably, the model converges rapidly on mobile devices, reaching 83.16% accuracy within six epochs, marking the first demonstration of near-full-precision performance in 4-bit training on general-purpose CPUs.

This work addresses the lack of systematic investigation into format selection and performance trade-offs in existing low-bit quantization-aware training (QAT) methods, as well as their insufficient evaluation on generative tasks. To this end, we propose the first integration of k-means clustering into QAT for 1-bit weight quantization, optimizing generative performance under a fixed inference memory budget. Our approach transcends the limitations of conventional integer-based quantization schemes by leveraging learned cluster centroids to better preserve model fidelity at ultra-low bitwidths. Experimental results demonstrate that, under identical memory constraints, our method significantly outperforms state-of-the-art integer quantization approaches while maintaining compatibility with general-purpose hardware for efficient deployment.

This work identifies and disentangles two orthogonal yet previously overlooked failure modes in low-bit floating-point quantization-aware training (HiF8 W8A8): silent forward representation corruption due to amax saturation and catastrophic forgetting induced by high learning rates—both undetectable through training loss. To mitigate these issues without additional supervision, the authors propose a conservative amax scaling strategy based on a 64-step historical window to suppress saturation, coupled with a 500-step BF16 warm-up phase followed by low-learning-rate QAT to prevent catastrophic forgetting. Evaluated on the OpenPangu-Embedded-1B model, this approach achieves near-lossless HiF8 quantization, with only 0.43% MMLU, 0.58% HellaSwag, and 0.22% ARC-Challenge performance degradation and a remarkably low average parameter error (APE) of 0.11% over 10,000 training steps.

This work addresses the severe degradation in reasoning performance of large language models under 4-bit quantization, particularly the loss of accuracy in low-entropy symbols such as digits and operators, which existing post-training quantization (PTQ) and quantization-aware training (QAT) methods struggle to recover. The authors propose ReQAT, a novel framework that identifies low-entropy tokens during inference as quantization-sensitive points and introduces three core techniques: Trace-Aligned QAT, Selective Entropy Minimization, and Quantization-Friendly Initialization (Q-FIT), collectively optimizing critical decision points. Combined with a RoPE-consistent KV cache transformation and enhancements to the FP4 format, ReQAT achieves higher accuracy than BF16 fine-tuning under full W4A4KV4 quantization and delivers up to 3.9× and 3.1× throughput speedups on NVIDIA DGX Spark and B200 systems, respectively, within the same training budget.

This work addresses the limited robustness of post-training quantization (PTQ) in the absence of target-task data by proposing a zero-shot transfer method. It introduces "quantization vectors" extracted via arithmetic operations in weight space and transfers them from a source model to a target Vision Transformer (ViT). This approach demonstrates for the first time that quantization robustness corresponds to a transferable direction in weight space, enabling significant improvements in resilience to low-bit quantization noise without requiring quantization-aware training, fine-tuning, or any data from the target task. Experiments show that the method can enhance PTQ robustness on ViTs by up to 60%, establishing a novel paradigm for low-cost, zero-shot quantization.