Revisiting Large Language Model Pruning using Neuron Semantic Attribution

2025-03-03

AI Summary

Existing large language model (LLM) pruning methods suffer from severe cross-task generalization bottlenecks—particularly catastrophic performance degradation on sentiment classification—and exhibit high sensitivity to calibration set selection. Method: We propose the first neuron-level semantic attribution framework for LLMs, integrating gradient- and activation-based signals to enable fine-grained, interpretable semantic characterization of unpruned neurons; we further design a neuron importance recalibration mechanism to enhance pruning robustness. Contribution/Results: Comprehensive evaluation across 24 datasets and four task categories demonstrates that our method substantially mitigates sentiment classification accuracy loss (average improvement of 12.7%) and enables semantic mapping of over 90% of critical neurons to concrete linguistic concepts. This work establishes, for the first time, an interpretable link between LLM pruning efficacy and the underlying semantic functionality of individual neurons.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Model pruning technique is vital for accelerating large language models by reducing their size and computational requirements. However, the generalizability of existing pruning methods across diverse datasets and tasks remains unclear. Thus, we conduct extensive evaluations on 24 datasets and 4 tasks using popular pruning methods. Based on these evaluations, we find and then investigate that calibration set greatly affect the performance of pruning methods. In addition, we surprisingly find a significant performance drop of existing pruning methods in sentiment classification tasks. To understand the link between performance drop and pruned neurons, we propose Neuron Semantic Attribution, which learns to associate each neuron with specific semantics. This method first makes the unpruned neurons of LLMs explainable.

Problem

• Evaluates pruning methods on diverse datasets and tasks
• Investigates impact of calibration set on pruning performance
• Proposes Neuron Semantic Attribution to explain pruned neurons

Innovation

• Neuron Semantic Attribution for explainable neurons
• Extensive evaluation across 24 datasets
• Calibration set impacts pruning performance