Two Sparse Matrices are Better than One: Sparsifying Neural Networks with Double Sparse Factorization

2024-09-27

arXiv.org

AI Summary

To address the low sparsification efficiency and severe performance degradation under high compression ratios in large language model (LLM) compression, this paper proposes Dual Sparse Factorization (DSF): each weight matrix is decomposed into two structured-sparse matrices, enabling joint reduction of both parameter count and computational cost. Unlike conventional one-shot pruning or low-rank approximation, DSF introduces a novel dual-sparse joint optimization paradigm, integrating an ADMM-driven alternating minimization algorithm with differentiable structured-sparsity constraints. On LLaMA2-13B, DSF achieves a single-pass 50% parameter reduction (halving model size) while outperforming the dense LLaMA2-7B in inference speed and accuracy. In CNN benchmarks, it significantly surpasses Optimal Brain Compression (OBC). Moreover, its accuracy advantage remains stable after fine-tuning. DSF establishes a new paradigm for high-ratio, high-fidelity neural network compression, balancing structural sparsity, trainability, and inference efficiency.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Data Compression
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Neural networks are often challenging to work with due to their large size and complexity. To address this, various methods aim to reduce model size by sparsifying or decomposing weight matrices, such as magnitude pruning and low-rank or block-diagonal factorization. In this work, we present Double Sparse Factorization (DSF), where we factorize each weight matrix into two sparse matrices. Although solving this problem exactly is computationally infeasible, we propose an efficient heuristic based on alternating minimization via ADMM that achieves state-of-the-art results, enabling unprecedented sparsification of neural networks. For instance, in a one-shot pruning setting, our method can reduce the size of the LLaMA2-13B model by 50% while maintaining better performance than the dense LLaMA2-7B model. We also compare favorably with Optimal Brain Compression, the state-of-the-art layer-wise pruning approach for convolutional neural networks. Furthermore, accuracy improvements of our method persist even after further model fine-tuning. Code available at: https://github.com/usamec/double_sparse.

Problem

• Reducing neural network size via double sparse factorization
• Achieving high sparsification while maintaining model performance
• Outperforming existing pruning methods like Optimal Brain Compression

Innovation

• Factorizes weight matrices into two sparse matrices
• Uses ADMM for efficient alternating minimization
• Achieves state-of-the-art neural network sparsification