LLM-Barber: Block-Aware Rebuilder for Sparsity Mask in One-Shot for Large Language Models

2024-08-20

arXiv.org

AI Summary

Existing LLM pruning methods overlook the dynamic drift of weight importance during pruning, leading to performance degradation. This work proposes a one-shot sparse mask reconstruction framework that requires no retraining. First, we introduce a novel block-level error propagation optimization mechanism to enable coordinated sparsification across Self-Attention and MLP modules. Second, we propose a gradient-weighted weight importance metric to dynamically capture importance drift throughout pruning. Third, we integrate block-aware error compensation with joint structural optimization to achieve true zero-shot retraining. Evaluated on LLaMA and OPT families (7B–13B), our method completes pruning in just 30 minutes on a single A100 GPU. It consistently outperforms state-of-the-art approaches in perplexity and zero-shot transfer tasks—without any fine-tuning or retraining.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large language models (LLMs) have grown significantly in scale, leading to a critical need for efficient model pruning techniques. Existing post-training pruning techniques primarily focus on measuring weight importance on converged dense models to determine salient weights to retain. However, they often overlook the changes in weight importance during the pruning process, which can lead to performance degradation in the pruned models. To address this issue, we present LLM-Barber (Block-Aware Rebuilder for Sparsity Mask in One-Shot), a novel one-shot pruning framework that rebuilds the sparsity mask of pruned models without any retraining or weight reconstruction. LLM-Barber incorporates block-aware error optimization across Self-Attention and MLP blocks, ensuring global performance optimization. Inspired by the recent discovery of prominent outliers in LLMs, LLM-Barber introduces an innovative pruning metric that identifies weight importance using weights multiplied by gradients. Our experiments show that LLM-Barber can efficiently prune models like LLaMA and OPT families with 7B to 13B parameters on a single A100 GPU in just 30 minutes, achieving state-of-the-art results in both perplexity and zero-shot performance across various language benchmarks. Code is available at https://github.com/YupengSu/LLM-Barber.

Problem

• Efficient pruning for large language models without retraining
• Block-aware error optimization in Self-Attention and MLP blocks
• Accurate weight importance identification using weights and gradients product

Innovation

• One-shot pruning without retraining or reconstruction
• Block-aware error optimization for global performance
• Weight-gradient product as pruning metric