FluxBin: Flexible LUT-based Ultra-low-bit LLM Inference by Algorithm-Kernel Synergy

2026-08-16

AI Summary

This study addresses the limited acceleration of binary quantization for large language models caused by the absence of specialized kernels and floating-point overhead. We propose an algorithm-kernel co-design framework that optimizes ultra-low-bit inference through decoupled row-column binary decomposition, Hessian-aware mixed bases, and lookup table operations with scale fusion via virtual column mapping. Experimental results demonstrate that this approach achieves a 5.92× speedup and 10.19× energy savings while maintaining accuracy comparable to fine-tuning methods. Furthermore, it reduces GPU memory footprint by 4× for 70B-parameter models, effectively unlocking the performance potential of binary quantization for single-GPU deployment.

Abstract

While binary quantization theoretically promises extreme compression and acceleration for Large Language Models (LLMs), existing research often overlooks the necessity of specialized hardware kernels, thus failing to unleash the full acceleration potential due to persistent reliance on expensive floating-point arithmetic or runtime dequantization overheads. To bridge this gap, we propose FluxBin (\textbf{F}lexible \textbf{L}UT-based \textbf{U}ltra-low-bit e\textbf{X}ecution with \textbf{Bin}ary bases), an algorithm-kernel co-design that synergizes post-training quantization with a highly optimized CUDA kernel. Algorithmically, we introduce Decoupled Row-Column Binary Decomposition to enhance representational capacity while maintaining hardware efficiency, complemented by a Hessian-guided saliency-aware hybrid bases that preserve critical information. At the kernel level, we implement a Lookup Table Building Approach with Scale Fusion to reduce floating-point arithmetic, featuring a Virtual Columnar Mapping that transforms irregular, sparse, and salient matrices into dense execution. Extensive evaluations demonstrate FluxBin achieves up to $5.92\times$ speedup and $10.19\times$ energy savings across diverse model architectures, delivering comparable accuracy to heavily fine-tuned methods. This effectively enables the deployment of 70B-scale models on one single A100 GPU with a $4\times$ memory reduction. Code is available at https://github.com/nicyyyy/FluxBin.

Problem

• Binary Quantization
• LLM Inference
• Hardware Kernels
• Ultra-low-bit
• Dequantization Overhead

Innovation

• Algorithm-Kernel Co-design
• Decoupled Row-Column Binary Decomposition
• Hessian-guided Saliency-aware Hybrid Bases
• Lookup Table with Scale Fusion
• Virtual Columnar Mapping