PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference

2026-07-16

AI Summary

This work addresses the challenge of achieving both accuracy and efficiency in low-bit quantization on CPUs, where existing methods are often hindered by coarse granularity or prohibitive overhead from mixed-precision schemes. Targeting edge CPUs, the authors co-design a compiler-aware quantization strategy that, given a user-specified average bit budget, employs activation-aware per-channel bit allocation. At compile time, channel clustering, reordering, and blocking transform fine-grained bit assignments into homogeneous block structures, enabling efficient SIMD/LUT-based kernels while eliminating runtime layout overhead. This approach realizes the first practical fractional-bit deployment on CPUs. Experiments show that under a 3-bit budget, it reduces perplexity by 2.4%–32.1% over prior methods, cuts activation reshuffling traffic by 70.8% through compiler optimizations, and achieves near-linear scaling of end-to-end latency and throughput with respect to the bit budget, with less than a 2% increase in per-token energy consumption.

Abstract

CPUs are the most universal target for on-device LLM inference, but existing low-bit quantization methods offer either coarse operating points or fine-grained mixed precision that is difficult to execute efficiently on CPUs. We present PolyQ, a CPU-oriented compiler/quantization co-design for activation-aware channel-wise bit allocation under a user-specified average-bit budget. PolyQ assigns per-channel bit-widths from $\{2,3,4,8,16\}$, then uses a compile-time model compiler to permute and cluster channels into bit-homogeneous blocks, generate SIMD- and LUT-compatible kernels, and merge compatible permutations across operators to keep layout regularization off the runtime path. This turns fine-grained budget fitting into a practical fractional-bit deployment method for CPU-only inference. Across Falcon-H1-3B, Llama2-13B, and Qwen3-32B on WikiText-2, PolyQ provides stable quality scaling from 3--6\,b and improves perplexity by 2.4--32.1\% over prior methods at a 3\,b target. End-to-end measurements on three representative CPUs -- workstation, laptop, and mobile -- show that compiler layout regularization reduces activation reorder traffic by up to 70.8\%, prefill latency and decode throughput scale nearly proportionally with the configured bit budget, and energy/token overhead stays below 2\% relative to an optimized LUT-based back-end. These results show that fractional-bit CPU deployment is practical, predictable, and energy-efficient across diverse edge targets.

Problem

• LLM inference
• low-bit quantization
• CPU
• edge computing
• mixed precision

Innovation

• quantization
• CPU inference
• compiler co-design
• channel-wise bit allocation
• SIMD optimization