Unifying Depth and Width Pruning for LLMs via Binary Knapsack Optimization

2026-08-13

AI Summary

Existing structured pruning methods for large language models rely on greedy strategies, which struggle to precisely meet compression budgets and suffer from myopic decision-making. This work proposes SNIPER, a two-stage pruning framework that first solves a binary knapsack optimization problem over coarse-grained components to obtain a conditionally optimal parameter allocation, followed by fine-grained pruning to strictly satisfy the target budget. The approach innovatively introduces the Compression Ratio Adherence Factor (CRAFT) to quantify budget fidelity and, for the first time, unifies knapsack-based optimization across both depth and width pruning, enabling near-exact compression control. Evaluated across four architectures and eighteen cross-domain tasks, SNIPER significantly outperforms six state-of-the-art methods, achieving superior average performance, enhanced stability, a CRAFT of 0.98, and an average ranking of 1.25.

Abstract

Structured pruning is a promising approach for compressing large language models (LLMs), yet existing methods rely heavily on greedy heuristics that produce myopic decisions, and often fail to precisely meet target compression budgets. We present SNIPER, a two-stage structured pruning framework that solves a knapsack optimization over coarse-granularity components to yield conditionally optimal parameter allocations with respect to fixed importance estimates, followed by a fine-grained pruning stage to meet strict budget constraints. We introduce the Compression Ratio Adherence Factor (CRAFT) to quantify budget fidelity, showing that while existing pruners deviate from target compression ratios by up to 33%, SNIPER achieves near-exact adherence with a CRAFT score of 0.98. Evaluations across four diverse architectures over a set of 18 tasks spanning five domains demonstrate SNIPER's consistent improvements in average performance retention and task-level stability over six state-of-the-art pruners. Across all pruning configurations, SNIPER achieves an excellent mean rank of 1.25, indicating its robust cross-architectural generalizability and excellent reliability.

Problem

• structured pruning
• large language models
• compression budget
• knapsack optimization
• model compression

Innovation

• structured pruning
• knapsack optimization
• compression budget adherence
• large language models
• CRAFT