SeVeDo: A Heterogeneous Transformer Accelerator for Low-Bit Inference via Hierarchical Group Quantization and SVD-Guided Mixed Precision

2025-12-14

AI Summary

Low-bit quantization reduces Transformer inference overhead but suffers from severe accuracy degradation due to activation outliers; existing outlier-handling methods preserve accuracy at the cost of high energy consumption. This work proposes a heterogeneous dual-path acceleration architecture: outlier-sensitive components are offloaded to a high-precision low-rank path, while the majority of computation is executed on a low-bit residual path. We introduce the first SVD-guided static mixed-precision allocation scheme and Hierarchical Group Quantization (HGQ), enabling accuracy-driven bit-width optimization and significantly reducing dequantization overhead. Experiments demonstrate a peak energy efficiency of 13.8 TOPS/W, achieving 12.7 TOPS/W on ViT-Base and 13.4 TOPS/W on Llama2-7B—substantially outperforming state-of-the-art low-bit accelerators.

Technology Category

• Machine Learning: Hardware-aware ML
• Computer Vision: Large Vision Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Low-bit quantization is a promising technique for efficient transformer inference by reducing computational and memory overhead. However, aggressive bitwidth reduction remains challenging due to activation outliers, leading to accuracy degradation. Existing methods, such as outlier-handling and group quantization, achieve high accuracy but incur substantial energy consumption. To address this, we propose SeVeDo, an energy-efficient SVD-based heterogeneous accelerator that structurally separates outlier-sensitive components into a high-precision low-rank path, while the remaining computations are executed in a low-bit residual datapath with group quantization. To further enhance efficiency, Hierarchical Group Quantization (HGQ) combines coarse-grained floating-point scaling with fine-grained shifting, effectively reducing dequantization cost. Also, SVD-guided mixed precision (SVD-MP) statically allocates higher bitwidths to precision-sensitive components identified through low-rank decomposition, thereby minimizing floating-point operation cost. Experimental results show that SeVeDo achieves a peak energy efficiency of 13.8TOPS/W, surpassing conventional designs, with 12.7TOPS/W on ViT-Base and 13.4TOPS/W on Llama2-7B benchmarks.

Problem

• Accelerates low-bit transformer inference efficiently
• Reduces energy consumption in quantization methods
• Handles activation outliers to maintain accuracy

Innovation

• Hierarchical Group Quantization reduces dequantization cost
• SVD-guided mixed precision minimizes floating-point operation cost
• Separates outlier-sensitive components into high-precision low-rank path