TRIM: Achieving Extreme Sparsity with Targeted Row-wise Iterative Metric-driven Pruning

2025-05-22

AI Summary

To address the severe performance degradation of large language models (LLMs) under high-sparsity pruning, this paper proposes a fine-grained, row-wise adaptive pruning method. Unlike conventional uniform intra-layer sparsity, our approach dynamically allocates heterogeneous sparsity ratios across output dimensions per layer, optimizing for minimal quality variance via iterative reweighting guided by gradient sensitivity and activation importance. This constitutes the first dimension-level, row-specific adaptive pruning strategy, breaking from layer-wise uniformity while remaining compatible with mainstream pruning frameworks and supporting end-to-end fine-tuning. Evaluated on Qwen2.5-14B and OPT-13B, our method achieves 80% sparsity with perplexity reductions of 48% and over 90%, respectively, alongside significantly improved zero-shot task stability—establishing new state-of-the-art results for extreme pruning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large Language Models (LLMs) present significant computational and memory challenges due to their extensive size, making pruning essential for their efficient deployment. Existing one-shot pruning methods often apply uniform sparsity constraints across layers or within each layer, resulting in suboptimal performance, especially at high sparsity ratios. This work introduces TRIM (Targeted Row-wise Iterative Metric-driven pruning), a novel approach that applies varying sparsity ratios to individual output dimensions (rows) within each layer. TRIM employs an iterative adjustment process guided by quality metrics to optimize dimension-wise sparsity allocation, focusing on reducing variance in quality retention across outputs to preserve critical information. TRIM can be seamlessly integrated with existing layer-wise pruning strategies. Our evaluations on perplexity and zero-shot tasks across diverse LLM families (Qwen2.5, LLaMA-2, and OPT) and sparsity levels demonstrate that TRIM achieves new state-of-the-art results and enhances stability. For instance, at 80% sparsity, TRIM reduces perplexity by 48% for Qwen2.5-14B and over 90% for OPT-13B compared to baseline methods. We conclude that fine-grained, dimension-wise sparsity adaptation is crucial for pushing the limits of extreme LLM compression. Code available at: https://github.com/flobk/TRIM

Problem

• Optimizes sparsity allocation in LLMs for efficient deployment
• Reduces performance degradation at high sparsity ratios
• Enhances model stability and critical information retention

Innovation

• Targeted row-wise iterative pruning method
• Varying sparsity ratios per output dimension
• Metric-driven iterative adjustment process