Structured Pruning of Large Language Models via Power Transformation and Sign-Preserving Score Aggregation with Adaptive Feature Retention

2026-07-08

AI Summary

This work addresses the challenges of transferring adaptive feature retention (AFR) from unstructured to structured pruning, which include heterogeneous pruning score distributions, loss of sign information, and outlier interference. To bridge this gap, the authors propose a unified structured pruning framework that introduces power transformation to align score distributions, designs a sign-preserving aggregation mechanism to maintain consistent optimization directions, and incorporates a percentile-based outlier removal strategy. This approach effectively narrows the performance gap between structured and unstructured pruning, achieving accuracy on Llama-3-8B, Vicuna-v1.5-13B, and LLaVA-v1.5-13B models that closely matches unstructured pruning while delivering substantial real-world inference speedups.

Abstract

This paper proposes an improved structured pruning method for large language models (LLMs) that addresses key challenges in adapting Adaptive Feature Retention (AFR), an unstructured pruning technique, to structured pruning. When applying AFR to structured pruning, three major problems arise: distribution mismatch between heterogeneous pruning scores, loss of sign information indicating optimization direction consistency, and influence of outliers. To address these issues, we propose a unified approach combining power transformation for nonlinear distribution alignment, sign-preserving score aggregation, and percentile-based outlier removal. Experiments on Llama-3-8B, Vicuna-v1.5-13B, and LLaVA-v1.5-13B demonstrate that our method maintains accuracy comparable to unstructured pruning while achieving practical inference speedup through structured pruning.

Problem

• structured pruning
• distribution mismatch
• sign information loss
• outlier influence
• large language models

Innovation

• structured pruning
• power transformation
• sign-preserving aggregation
• adaptive feature retention
• outlier removal