Quant.npu: Enabling Efficient Mobile NPU Inference for on-device LLMs via Fully Static Quantization

2026-05-19

AI Summary

This work addresses the challenge that existing post-training quantization methods, which rely on dynamic activation quantization, fail to meet the fully static quantization constraints imposed by mobile NPUs, thereby hindering efficient on-device deployment of large language models. To overcome this limitation, we propose Quant.npu, an integer-only, fully static quantization framework that enables low-bitweight quantization through learnable quantization parameters and weight rotation, while eliminating runtime recomputation via a two-stage pipeline. Key innovations include rotation- and bitwidth-aware initialization, distribution-aware selective optimization, and a sensitivity-guided adaptive mixed-precision strategy. Experiments demonstrate that Quant.npu achieves state-of-the-art accuracy on real mobile NPUs while reducing inference latency by up to 15.1%.

Abstract

Large language models (LLMs) are increasingly deployed on mobile devices, where Neural Processing Units (NPUs) necessitate fully static quantization for optimal inference efficiency. However, existing post-training quantization (PTQ) methods predominantly rely on dynamic activation quantization, rendering them incompatible with NPU hardware constraints. To bridge the gap between high-fidelity PTQ and NPU-constrained inference, we propose Quant.npu, a integer-only fully static quantization framework. It incorporates learnable quantization parameters and rotation matrices, enabling low-bit activation-weight quantization without runtime quantization parameters re-computation. Crucially, we identify that initialization and selective optimization of quantization parameters is pivotal for optimization stability, as improper initialization and naive joint optimization induce gradient instability that disrupts the optimization of rotation matrices. To address this, we propose a rotation-and-bit-width-aware initialization tailored to diverse activation profiles and a distribution-aware selective optimization (two-stage quantization pipeline) tailored to rotated and unrotated tensors. Furthermore, we introduce a sensitivity-guided adaptive mixed-precision scheme to balance accuracy with inference efficiency. Extensive experiments on real-world mobile NPUs demonstrate that Quant.npu achieves comparable accuracy to state-of-the-art methods, while reducing inference latency by up to 15.1%.

Problem

• fully static quantization
• mobile NPU
• on-device LLMs
• post-training quantization
• hardware constraints

Innovation

• fully static quantization
• NPU inference
• rotation-aware initialization
• selective optimization
• adaptive mixed-precision