UnionSparse: An Index-Efficient Sparsity Framework for Low-Bit Sparse LLM Inference on Edge

2026-08-10

AI Summary

This work addresses the inefficiency of low-bit sparse large language model inference on edge devices, which is hindered by the overhead of sparse metadata. To tackle this challenge, the authors propose UnionSparse, a framework that introduces the Payload-to-Metadata Ratio as a key metric and integrates two core innovations: Index-Efficient Bitmap Encoding (IE-BME) for compact metadata representation and Low-Bit Shared-Memory Parallel Decoding (LSPD) for efficient computation. Combined with W4A4 quantization and a customized sparse matrix-matrix multiplication (SpMM) kernel, the framework jointly optimizes index compression and computational load. Experimental results demonstrate that UnionSparse achieves speedups of 2.30×, 1.43×, 1.56×, and 3.46× over FlashLLM, SpInfer, CUTLASS, and cuBLAS, respectively, across sparsity levels ranging from 30% to 70%, significantly enhancing sparse inference efficiency on edge devices.

Abstract

Edge LLM inference combines sparsity and low-bit quantization to meet device memory, latency, and power limits. Yet quantization shrinks weight payloads without proportionally reducing sparse metadata, so index traffic and nonzero extraction become critical SpMM bottlenecks. We introduce the Payload-to-Metadata Ratio (PMR) and show that improving PMR raises effective compute intensity in decoding. We present UnionSparse, an index-efficient framework that combines Index-Efficient Bitmap Encoding (IE-BME) with a SpMM kernel using Low-Bit Shared-Memory Parallel Decoding (LSPD). IE-BME amortizes metadata and aligns sparse traversal with fragment assembly, while LSPD improves small-batch execution. Under W4A4 quantization and 30%--70% sparsity, UnionSparse outperforms FlashLLM and SpInfer by 2.30x and 1.43x, and CUTLASS and cuBLAS Tensor Core by 1.56x and 3.46x, respectively. These results establish payload-extraction efficiency as a first-order concern for low-bit sparse inference on edge GPUs. Source code is available at: https://github.com/Victor-Alen/UnionSparse.

Problem

• sparse LLM inference
• low-bit quantization
• index overhead
• SpMM bottleneck
• edge computing

Innovation

• UnionSparse
• Payload-to-Metadata Ratio
• Index-Efficient Bitmap Encoding
• Low-Bit SpMM
• Edge LLM Inference