Quantization Inflates Reasoning: Token Inflation as a Hidden Cost of Low-Bit Reasoning Models

2026-06-24

AI Summary

This study addresses an underappreciated drawback of low-bit quantization in large language models: chain-of-thought (CoT) token inflation, wherein quantized models generate significantly more reasoning tokens during inference, thereby offsetting expected speedup gains. The work introduces the CoT Token Inflation Ratio to systematically quantify this phenomenon and demonstrates that even when answer accuracy remains stable, INT4 and INT3 quantization substantially increase reasoning length. Through extensive experiments across mathematical reasoning, code generation, scientific question answering, and agent-based tasks, the authors establish the ubiquity of token inflation. They further show that quantization-aware training effectively mitigates both accuracy degradation and token inflation. The study advocates for incorporating inference length into the evaluation framework for quantized models to enable a more holistic assessment of real-world deployment performance.

Abstract

Quantization is widely used to reduce the inference cost of large language models, but its effect on reasoning models is not fully captured by final-answer accuracy or per-token latency. We show that low-bit post-training quantization can introduce a hidden test-time compute cost: quantized reasoning models often generate longer chains of thought even when they still answer correctly. Across mathematical reasoning, code generation, scientific question answering, and agentic tool-use benchmarks, we find that INT4/INT3 quantization can preserve accuracy but increase reasoning-token usage, offsetting the expected per-token speedup. To measure this effect, we introduce the CoT Token Inflation Ratio, which compares reasoning length between quantized and full-precision models averaged across all evaluation benchmarks. We further show that token inflation is accompanied by behavioral changes in the reasoning trace, including more intermediate steps and greater semantic repetition. These changes translate into measurable end-to-end real-world serving penalties. Finally, we evaluate mitigation strategies and find that prompting and decoding-time sampling offer inconsistent accuracy-length trade-offs, while quantization-aware training shows more promise in reducing both accuracy degradation and token inflation. Our results suggest that reasoning-token usage should be reported alongside accuracy when evaluating quantized reasoning models.

Problem

• quantization
• reasoning models
• token inflation
• chain-of-thought
• inference cost

Innovation

• token inflation
• quantization
• chain-of-thought
• reasoning efficiency
• quantization-aware training