QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstruction

2026-08-14

AI Summary

This study addresses the elevated loss lower bounds caused by reconstruction errors and performance degradation in low-bit models during Quantization-Aware Training (QAT). We propose Quasar, a method that seamlessly integrates lightweight loss-aware reconstruction into the QAT loop for the first time. By leveraging online saliency estimation and affine dequantizer fitting, Quasar establishes reconstruction as a critical convergence objective without incurring inference overhead. Experimental results demonstrate that Quasar achieves optimal KL divergence at 2–4 bits and improves 2-bit accuracy by 3.5–4.3 percentage points over existing baselines. These findings indicate that Quasar significantly outperforms current methods, effectively overcoming performance bottlenecks in low-bit large language models.

Abstract

As large language model inference shifts toward lower precision, post-training quantization (PTQ) becomes increasingly brittle, making quantization-aware training (QAT) essential for preserving model quality. However, QAT computes the loss and surrogate gradients using a lossy reconstruction of latent full-precision weights, while applying updates to the latent weights themselves. This mismatch can lead to suboptimal training trajectories and a higher loss floor. Second-order PTQ methods mitigate a similar gap by minimizing loss-aware reconstruction error, but doing it once for a frozen model can take hours; repeating this process throughout QAT as the weights evolve is impractical. We introduce QUASAR, a QAT method that continuously performs lightweight, loss-aware reconstruction in the training loop to lower the loss floor and improve the resulting low-bit model. At each training step, QUASAR uses the exponential moving average of squared gradients as online saliency estimates, searches over a small set of clipping ranges, and fits affine dequantizers via saliency-weighted least squares. Our analysis shows that the loss-aware reconstruction error is the only reconstruction-dependent term in the QAT convergence bound and controls the loss of the final quantized model, establishing QUASAR's objective as a principled optimization target. QUASAR modifies only the training procedure and supports standard deployment formats, including integer quantization and NVFP4, with no inference-time changes or overhead. Across Qwen3 and Llama-3.1, QUASAR achieves the lowest held-out KL divergence among competitive QAT methods at 2, 3, and 4 bits, reducing KL by at least 10% at 3 and 4 bits and by 29% at 2 bits. At 2 bits, it improves average accuracy across eight tasks by 3.5-4.3 percentage points over strong QAT and PTQ baselines.

Problem

• Quantization-Aware Training
• Loss Floor
• Large Language Models
• Low-bit Quantization
• Reconstruction Error

Innovation

• Quantization-Aware Training
• Loss-Aware Reconstruction
• Online Saliency Estimation
• Low-Bit Quantization
• Affine Dequantizer