Reliability Scaling Laws for Quantized Large Language Models

2026-07-12

AI Summary

This study addresses the unclear reliability of quantized large language models under perturbed inputs, a key barrier to their real-world deployment. The authors systematically evaluate six quantization methods at 2-, 3-, 4-, and 8-bit precisions in terms of uncertainty estimation, calibration, and robustness, leveraging character- and word-level semantic-preserving perturbations alongside calibration analysis techniques. Their findings reveal, for the first time, a nonlinear relationship between quantization bit-width and model reliability, demonstrating that 4-bit models achieve an optimal trade-off between efficiency and reliability. Furthermore, the work provides empirical evidence that quantization substantially enhances model robustness against natural input perturbations.

Abstract

Quantization is a powerful strategy to build capable and resource-efficient large language models (LLMs) by reducing the bitwidth of the parameters. While quantized LLMs achieve state-of-the-art performance on unperturbed inputs using standard predictive metrics, their performance on perturbed inputs, measured using reliability metrics, remains underexplored, despite its importance for reliable deployment. To address this gap, we first conduct a comprehensive reliability evaluation of quantized LLMs consisting of three key components: (1) Uncertainty: We assess the trustworthiness of LLMs quantized to 2, 3, 4, and 8 bits using six different quantization methods, employing established uncertainty metrics. (2) Calibration: We assess how well-calibrated the uncertainty estimates of quantized models are across model scales and bit precisions. (3) Robustness: We design character-level and word-level input perturbations to evaluate the reliability of quantized models under semantically-preserving variations in the inputs that arise in real-world applications. Second, we characterize how reliability scales with the total number of model bits. Our study reveals that while the performance scales monotonically with the total number of bits, the reliability scalings are nonlinear. A reliability peak occurs for 4-bit quantized models, indicating that quantizing moderately sized models offers the best reliability-efficiency trade-off. Additionally, our empirical findings reveal that quantization enhances the robustness of LLMs to natural input perturbations.

Problem

• quantization
• large language models
• reliability
• robustness
• uncertainty

Innovation

• Reliability Scaling Laws
• Quantized LLMs
• Uncertainty Calibration
• Input Robustness
• Bitwidth Efficiency