The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs

2026-07-09

AI Summary

Current quantitative evaluations of large language models rely primarily on accuracy and perplexity, which often fail to capture behavioral changes induced by quantization. This work proposes a novel metric—correctness consistency—that measures the overlap in correct predictions between the original and quantized models at the decision level. Complementing this with an analysis of structural perturbations in attention weights, the study introduces, for the first time, a decision-level behavioral consistency measure. Experiments across 2-bit to 8-bit quantization reveal that even when task performance appears preserved, moderate quantization levels already induce significant behavioral divergence. Notably, query and key projections exhibit heightened sensitivity to quantization, exposing an “equivalence illusion” wherein conventional performance metrics mask underlying discrepancies in model behavior.

Abstract

Post-training quantization is widely used to deploy large language models in resource-constrained settings, yet its evaluation relies almost exclusively on accuracy and perplexity. We show that these metrics fail to capture behavioral changes induced by quantization. We introduce correctness agreement, a decision-level metric that measures overlap in correct predictions between a base model and its quantized variants, independent of absolute accuracy. Across multiple models and quantization schemes from 8-bit to 2-bit, we find that behavioral divergence emerges under moderate quantization even when task performance appears preserved. To explain this effect, we analyze quantization as a structural operator on attention weights and quantify layer-wise distortions using statistical and distributional measures. Our results reveal non-linear breakpoints at low bit-widths and show that query and key projections are consistently more sensitive than value and output projections. These findings expose an illusion of equivalence between base and quantized models and motivate behavioral evaluation beyond conventional performance metrics.

Problem

• quantization
• large language models
• behavioral divergence
• evaluation metrics
• attention weights

Innovation

• correctness agreement
• post-training quantization
• behavioral divergence
• attention weight distortion
• quantization sensitivity