Mixed-Precision Quantization: Make the Best Use of Bits Where They Matter Most

2024-12-04

arXiv.org

AI Summary

This study addresses the optimal bit-width allocation problem in mixed-precision quantization. Methodologically, it introduces a novel search-based framework featuring two variants of particle swarm optimization: Penalty-Driven PSO (PPSO) and Greedy-Criterion PSO (GC-PSO), both rigorously modeling integer constraints and proven convergent via dynamical systems theory. The key contribution is the first unified bit-allocation paradigm spanning diverse domains—digital signal processing (FIR filters), wireless communications (receivers), and machine learning (gradient descent). Experimental results demonstrate that the proposed approach achieves superior trade-offs between quantization accuracy and hardware resource efficiency, significantly outperforming state-of-the-art quantization methods across benchmarks.

Technology Category

• Search and Optimization: Mixed Discrete/Continuous Search
• Machine Learning: Optimization
• Constraint Satisfaction and Optimization: Mixed Discrete/Continuous Optimization

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Mixed-precision quantization offers superior performance to fixed-precision quantization. It has been widely used in signal processing, communication systems, and machine learning. In mixed-precision quantization, bit allocation is essential. Hence, in this paper, we propose a new bit allocation framework for mixed-precision quantization from a search perspective. First, we formulate a general bit allocation problem for mixed-precision quantization. Then we introduce the penalized particle swarm optimization (PPSO) algorithm to address the integer consumption constraint. To improve efficiency and avoid iterations on infeasible solutions within the PPSO algorithm, a greedy criterion particle swarm optimization (GC-PSO) algorithm is proposed. The corresponding convergence analysis is derived based on dynamical system theory. Furthermore, we apply the above framework to some specific classic fields, i.e., finite impulse response (FIR) filters, receivers, and gradient descent. Numerical examples in each application underscore the superiority of the proposed framework to the existing algorithms.

Problem

• Optimize bit allocation in mixed-precision quantization systems
• Develop efficient algorithms for constrained integer bit allocation
• Apply framework to FIR filters, receivers, and gradient descent

Innovation

• Proposes a new bit allocation framework
• Introduces penalized particle swarm optimization
• Develops greedy criterion PSO algorithm