🤖 AI Summary
This work addresses the challenge of efficiently compressing large language models without training while minimizing performance degradation. The authors propose a training-free compression method that formulates layer-wise sparsity allocation as a multiple-choice knapsack problem, dynamically distributing sparsity across layers under a global compression budget using calibration data. Efficient weight reconstruction is achieved via a single-step sparse matrix factorization inspired by dictionary learning, circumventing iterative optimization and backpropagation to drastically reduce computational overhead. Experiments demonstrate superior performance over existing methods at compression rates of 20%–50%, with over 90% of the original model’s performance retained at 30% compression. Moreover, after light fine-tuning—e.g., compressing Qwen3-14B to 8B and refining with 30M tokens—the model nearly matches the performance of the native Qwen3-8B.
📝 Abstract
We present ROCKET, a training-free model compression method that achieves state-of-the-art performance in comparison with factorization, structured-sparsification and dynamic compression baselines. Operating under a global compression budget, ROCKET comprises two key innovations: First, it formulates layer-wise compression allocation as a multi-choice knapsack problem, selecting the optimal compression level for each layer to minimize total reconstruction error while adhering to a target model size. Second, it introduces a single-step sparse matrix factorization inspired by dictionary learning: using only a small calibration set, it sparsifies weight coefficients based on activation-weights sensitivity and then updates the dictionary in closed form via least squares bypassing iterative optimization, sparse coding, or backpropagation entirely. ROCKET consistently outperforms existing compression approaches across different model architectures at 20-50\% compression rates. Notably, it retains over 90\% of the original model's performance at 30\% compression without any fine-tuning. Moreover, when applying a light fine-tuning phase, recovery is substantially enhanced: for instance, compressing Qwen3-14B to an 8B-parameter model and healing it with just 30 million tokens yields performance nearly on par with the original Qwen3-8B. The code for ROCKET is at github.com/mts-ai/ROCKET/tree/main.