FlashQuant: Sparse-Dense Fusion for Memory-Efficient Outlier-Aware LLM Inference

2026-08-16

AI Summary

This study addresses the memory redundancy and decoding inefficiency caused by separate sparse and dense execution paths in outlier-aware quantization for large language models. We propose FlashQuant, a novel framework featuring a fused sparse-dense execution mechanism that integrates GEMM and SpMM operations into a single kernel for W4A16 decoding. By leveraging Tile-COO encoding, tiling strategies, and pipeline scheduling, FlashQuant enables on-chip sharing of activation and output tiles across heterogeneous computations, thereby eliminating redundant memory accesses and overlapping computation with data transfer. Experimental results demonstrate that this framework significantly reduces outlier handling overhead, achieving a 2.74–4.18× speedup over cuBLAS BF16 and outperforming the strongest unfused baseline by 1.53×.

Abstract

Low-bit quantization reduces the memory footprint and computational cost of large language model (LLM) inference. However, high-magnitude outlier weights can induce substantial quantization errors and degrade model accuracy. Outlier-aware quantization addresses this issue by retaining outliers in high precision while quantizing the remaining weights, resulting in a low-bit dense GEMM path and a high-precision sparse SpMM path. Existing implementations execute these paths in separate GPU kernels, despite their shared activations and outputs, thereby missing opportunities for intra-operator reuse and incurring redundant global-memory accesses. This inefficiency is particularly pronounced in memory-bound decoding workloads. We propose FlashQuant, a content-sharing execution framework for outlier-aware W4A16 decoding. FlashQuant fuses the dense GEMM and sparse outlier SpMM paths into a single GPU kernel, enabling on-chip reuse of activation and output tiles across heterogeneous computations. It introduces three key techniques: sparse-dense tiling, which aligns outlier processing with dense GEMM tiles; Tile-COO outlier encoding, which enables efficient sparse access and reduces shared-memory bank conflicts; and pipelined scheduling, which overlaps computation with data movement. Experiments show that FlashQuant reduces outlier-processing overhead, achieving $2.74\times - 4.18\times$ speedup over cuBLAS BF16 and up to $1.53\times$ speedup over the strongest unfused outlier-aware baseline.

Problem

• Outlier-Aware Quantization
• LLM Inference
• Memory Efficiency
• Kernel Fusion
• Sparse-Dense Computation

Innovation

• Sparse-Dense Fusion
• Outlier-Aware Quantization
• Tile-COO Encoding
• Memory-Efficient Inference
• Kernel Fusion