Problem
Research questions and friction points this paper is trying to address.
extreme compression
efficient inference
systolic-array-based accelerators
vector quantization
input-dependent sparsity
Vortex: Bridging Extreme Compression and Efficient LLM Inference
๐ค AI Summary
็ ็ฉถ้่ฟVortexๆถๆ่งฃๅณๆ็ซฏๅ็ผฉๆๆฏๅจๅฎ้
ๅบ็จไธญ็ๆ็้ฎ้ข๏ผ้็จๅๆตๆง่ก็ญ็ฅๅไปฃ็ ๆฌไธไธๆ็จ็ๆงไผๅ๏ผๅฎ็ฐ้ซๆๆจ็ใ
๐ Abstract
Extreme compression techniques, including vector quantization (VQ) and input-dependent sparsity, can significantly reduce the memory footprint of large language models (LLMs). However, a key challenge remains in translating such compression into practical efficiency. On conventional systolic-array-based accelerators, VQ incurs high dequantization overhead, while the irregular patterns of input-dependent sparsity are difficult to exploit. In this study, we address these challenges with Vortex, an architecture compatible with systolic-array-based accelerators with minimal hardware overhead, bridging the gap between extreme compression and efficient inference. Vortex adopts a bi-flow execution strategy that efficiently supports vector-quantized models across both prefill and decoding workloads, and we further optimize it through systematic design space exploration. On the algorithm side, we propose codebook-wise contextual sparsity to align with VQ execution. Across end-to-end workloads, Vortex achieves $8.03\times$--$23.7\times$ speedup and $5.68\times$--$12.5\times$ energy reduction over state-of-the-art accelerators.
Innovation
Methods, ideas, or system contributions that make the work stand out.
bi-flow execution strategy
vector-quantized models
codebook-wise contextual sparsity
systolic-array-based accelerators
๐ Similar Papers
