🤖 AI Summary
To address the rigidity of low-rank weight approximations and substantial accuracy degradation in post-training compression of large language models (LLMs), this paper proposes CoSpaDi—a training-free LLM compression framework that requires no fine-tuning. Its core innovation replaces conventional low-rank decomposition with sparse dictionary learning: using a small calibration dataset, it jointly optimizes a dense dictionary and a column-wise sparse coefficient matrix, enabling flexible, structured sparse representations of weight columns in heterogeneous subspaces. Additionally, an output activation matching strategy is introduced to preserve functional fidelity. CoSpaDi natively supports synergistic sparse computation and quantization. Experiments on Llama and Qwen families demonstrate that CoSpaDi consistently outperforms state-of-the-art low-rank methods across 20%–50% compression ratios, achieving lower perplexity and higher task accuracy.
📝 Abstract
Post-training compression of large language models (LLMs) largely relies on low-rank weight approximation, which represents each column of a weight matrix in a shared low-dimensional subspace. While this is a computationally efficient strategy, the imposed structural constraint is rigid and can lead to a noticeable model accuracy drop. In this work, we propose CoSpaDi (Compression via Sparse Dictionary Learning), a novel training-free compression framework that replaces low-rank decomposition with a more flexible structured sparse factorization in which each weight matrix is represented with a dense dictionary and a column-sparse coefficient matrix. This formulation enables a union-of-subspaces representation: different columns of the original weight matrix are approximated in distinct subspaces spanned by adaptively selected dictionary atoms, offering greater expressiveness than a single invariant basis. Crucially, CoSpaDi leverages a small calibration dataset to optimize the factorization such that the output activations of compressed projection layers closely match those of the original ones, thereby minimizing functional reconstruction error rather than mere weight approximation. This data-aware strategy preserves better model fidelity without any fine-tuning under reasonable compression ratios. Moreover, the resulting structured sparsity allows efficient sparse-dense matrix multiplication and is compatible with post-training quantization for further memory and latency gains. We evaluate CoSpaDi across multiple Llama and Qwen models under per-layer and per-group settings at 20-50% compression ratios, demonstrating consistent superiority over state-of-the-art data-aware low-rank methods both in accuracy and perplexity. Our results establish structured sparse dictionary learning as a powerful alternative to conventional low-rank approaches for efficient LLM deployment.