Beyond Layer Importance in Layer-wise Sparsity: An Inter-Layer Perturbation-Absorption Perspective

2026-06-13

AI Summary

Existing layer-wise pruning methods for large language models rely solely on local importance metrics, neglecting the cross-layer perturbation compensation mechanism. This work reveals through controlled perturbation experiments that early layers amplify perturbations while middle-to-late layers actively absorb them. For the first time, the study characterizes this compensatory behavior from a “perturbation–absorption” perspective and introduces a layer absorption coefficient to quantify absorption capacity. Building on this insight, the authors propose an absorption-aware correction strategy to guide pruning resource allocation. When integrated with OWL and AlphaPruning at 70% sparsity, the approach reduces perplexity by 7.13% and improves zero-shot accuracy by 1.02%.

Abstract

The considerable layer-wise redundancy in large language models (LLMs) has established non-uniform sparsity allocation across layers as the standard pruning approach for efficient compression. Existing layer-wise allocation methods that estimate allocation strategy from local signals such as activation outliers or weight spectra mainly derive from local layer importance, whereas the final post-pruning performance is also influenced by the network's subsequent compensatory capacity. In this paper, we directly characterize this property through controlled perturbation experiments. We make the following empirical findings. First, layers exhibit highly heterogeneous responses to pruning-scale perturbations. In most cases, early layers amplify perturbations, while middle and late layers actively absorb them, with relative L2 drift decreasing monotonically across depth and direction realigning toward the unperturbed hidden-state trajectory. Second, absorption is a large-perturbation phenomenon. Under small perturbations the network exhibits amplification across all layers, and the transition to absorption occurs smoothly as perturbation magnitude grows to pruning scale. This enriches the linearized accumulation theory underlying related works. Building on these findings, we define an absorption coefficient per layer and propose absorption-aware correction, an orthogonal augmentation that improves OWL and AlphaPruning by reducing perplexity by 7.13% and boosting zero-shot accuracy by 1.02% across multiple model families at 70% sparsity.

Problem

• layer-wise sparsity
• pruning
• perturbation absorption
• large language models
• compensatory capacity

Innovation

• inter-layer perturbation absorption
• layer-wise sparsity allocation
• absorption coefficient
• pruning compensation
• large language model compression