QuantuneV2: Compiler-Based Local Metric-Driven Mixed Precision Quantization for Practical Embedded AI Applications

2025-01-13

AI Summary

Existing mixed-precision quantization methods for embedded AI deployment rely on costly retraining and overlook compiler-induced computational overhead and intermediate state expansion, leading to accuracy degradation and increased inference latency. This paper proposes a compiler-level post-training mixed-precision quantization framework. It introduces the first stable sensitivity analysis based on local metrics—including weight and activation distributions, signal-to-quantization-noise ratio (SQNR), and mean squared error (MSE)—to guide precision assignment. The framework jointly optimizes intermediate representation selection and operator fusion, enabling efficient quantization decisions entirely at compile time. It requires only two forward passes with linear time complexity and eliminates the need for model retraining. Evaluated on five models including ResNet18v1, our method achieves up to 10.28% higher accuracy and 12.52% faster inference compared to state-of-the-art approaches, demonstrating synergistic optimization of accuracy and latency.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Learning to Search
• Computer Vision: Learning & Optimization for CV

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Security and Privacy: Large-scale security measurements

Abstract

Mixed-precision quantization methods have been proposed to reduce model size while minimizing accuracy degradation. However, existing studies require retraining and do not consider the computational overhead and intermediate representations (IR) generated during the compilation process, limiting their application at the compiler level. This computational overhead refers to the runtime latency caused by frequent quantization and dequantization operations during inference. Performing these operations at the individual operator level causes significant runtime delays. To address these issues, we propose QuantuneV2, a compiler-based mixed-precision quantization method designed for practical embedded AI applications. QuantuneV2 performs inference only twice, once before quantization and once after quantization, and operates with a computational complexity of O(n) that increases linearly with the number of model parameters. We also made the sensitivity analysis more stable by using local metrics like weights, activation values, the Signal to Quantization Noise Ratio, and the Mean Squared Error. We also cut down on computational overhead by choosing the best IR and using operator fusion. Experimental results show that QuantuneV2 achieved up to a 10.28 percent improvement in accuracy and a 12.52 percent increase in speed compared to existing methods across five models: ResNet18v1, ResNet50v1, SqueezeNetv1, VGGNet, and MobileNetv2. This demonstrates that QuantuneV2 enhances model performance while maintaining computational efficiency, making it suitable for deployment in embedded AI environments.

Problem

• Mixed-Precision Quantization
• Re-training Efficiency
• Compilation Overhead

Innovation

• QuantuneV2
• Hybrid-Precision Quantization
• Embedded AI Optimization