ARCHead: Activation-Metric Residual Correction for Large Language Model Output Heads

2026-08-03

AI Summary

This work addresses the significant storage overhead incurred by retaining the language model head (LM-head) in high-precision formats after weight quantization, as direct quantization severely distorts the logit distribution. To overcome this challenge, the authors propose ARCHead, a novel LM-head compression method that introduces, for the first time, an activation-aware metric combined with low-rank decomposition, grouped INT4 residual quantization, and a low-rank correction mechanism. This approach achieves substantial storage reduction with nearly negligible performance degradation. Evaluated on Qwen3-8B-Base, ARCHead requires only 25.6% of the storage of a BF16 head while achieving a relative perplexity of 1.007. When replacing the original BF16 head, it incurs a minimal cross-entropy increase of merely 0.006–0.007 and reduces throughput by less than 2%.

Abstract

Weight-only quantization substantially reduces the storage of large language model (LLM) transformer blocks, but practical backends often retain the final language-modeling head (LM-head) in BF16 or FP16. Quantizing this projection naively can strongly perturb the vocabulary-logit distribution. We present ARCHead, a packed LM-head compressor that combines a quantized low-rank core, group-wise INT4 residuals, and a low-rank correction fitted in an activation-derived metric. ARCHead stores no dense BF16 head and reduces persistent LM-head storage by 3.7-3.9x. On Qwen3-8B-Base, it uses 25.6% of BF16 head storage while attaining 1.007 relative perplexity; storage-matched naive INT4 yields 1.14-1.16. Replacing the BF16 head left by AWQ or bitsandbytes adds only 0.006-0.007 cross-entropy, with less than 2% throughput change in our measurements. ARCHead therefore complements block quantizers by compressing the large output projection they can leave untouched. Code is available at https://github.com/suayptalha/archead.

Problem

• weight-only quantization
• language modeling head
• logit distribution
• low-bit quantization
• storage compression

Innovation

• weight-only quantization
• low-rank correction
• activation-derived metric
• LM-head compression
• INT4 residuals