Reservoir of Importance: Learning Semi-Structured Sparsity with Differentiable Subset Sampling

2026-08-24

AI Summary

本文提出了一种轻量级的半结构化剪枝框架RoI，通过可微子集采样学习稀疏掩码，减少参数和内存开销，实现高效的大语言模型加速。

Abstract

Semi-structured $N$:$M$ sparsity has emerged as a practical direction for accelerating large language models (LLMs). However, existing learnable-mask approaches incur substantial parameter and memory overhead, limiting their scalability to large models and aggressive sparsity regimes. In this work, we revisit semi-structured pruning from a perspective that reconciles efficiency with scalability. We propose Reservoir of Importance (RoI), a lightweight semi-structured pruning framework that learns sparsity masks through differentiable subset sampling. Unlike prior methods that model full categorical distributions over all feasible $N$:$M$ patterns, RoI introduces a compact-logit parameterization for sparsity mask learning and performs sampling without replacement to select masks, thereby reducing trainable parameters from combinatorial complexity to $\mathcal{O}({M})$. As a result, RoI requires 1.5-8.75$\times$ fewer learnable parameters and significantly lower memory cost, while remaining fully aligned with hardware-friendly sparsity patterns. Extensive evaluations across multiple scales of the Qwen2.5 LLM family (0.5-7B parameters) demonstrate that RoI achieves competitive performance with strong memory efficiency, stability, and scalability to more aggressive $N$:$M$ sparsity patterns, offering a practical path toward efficient LLM deployment.

Problem

• Semi-structured Sparsity
• Large Language Models
• Parameter Overhead
• Memory Overhead
• Scalability

Innovation

• Semi-structured Pruning
• Differentiable Subset Sampling
• Compact-logit Parameterization