EPTS: Elastic Post-Training Sparsity for Efficient Large Language Model Compression

2026-06-23

AI Summary

This work addresses the inflexibility of existing post-training sparsification methods, which support only a single sparsity level and require re-optimization for each hardware deployment scenario. To overcome this limitation, we propose Elastic Post-Training Sparsification (EPTS), a framework that enables a single optimization pass to deliver high performance across diverse sparsity configurations. EPTS introduces two key innovations: Multi-Sparsity Hierarchical LoRA (MS-HiLoRA), which facilitates knowledge transfer from low to high sparsity levels, and a Multi-Sparsity Feature Mixer (MSFM) that dynamically fuses multi-granularity sparse features to enhance model robustness. Experiments on large language models such as LLaMA and OPT demonstrate that EPTS achieves accuracy comparable to state-of-the-art methods like SparseGPT and Wanda, while substantially improving deployment efficiency—enabling “optimize once, deploy everywhere” across varied sparsity-aware hardware environments.

Abstract

Post-Training Sparsity (PTS) has emerged as a crucial paradigm for compressing Large Language Models to facilitate efficient deployment on resource-constrained devices. However, existing PTS methodologies are typically confined to Single-Sparsity optimization, necessitating a separate, time-consuming optimization session for each specific sparsity level. This rigid paradigm significantly hinders flexible deployment across diverse hardware scenarios, as adapting to a new sparsity requirement mandates a complete re-optimization process. To address these limitations, we propose Elastic Post-Training Sparsity (EPTS), a unified Multi-Sparsity framework that produces a single elastic model capable of maintaining robust performance across diverse sparsity configurations through a one-shot optimization process. Specifically, we design a Multi-Sparsity Hierarchy LoRA (MS-HiLoRA) mechanism that facilitates knowledge inheritance from low- to high-sparsity groups, effectively mitigating the competition for parameter reconstruction. Furthermore, we introduce a Multi-Sparsity Feature Mixer (MSFM), which significantly enhances the model's adaptability to pruning perturbations by dynamically fusing feature representations of varying sparsity granularities. Extensive experiments on LLaMA and OPT families demonstrate that EPTS achieves competitive performance compared to state-of-the-art methods like SparseGPT and Wanda, while offering significant efficiency gains by enabling multi-scenario deployment from a single optimization. our source code is available at https://github.com/xuke225/EPTS.

Problem

• Post-Training Sparsity
• Multi-Sparsity
• Model Compression
• Large Language Models
• Efficient Deployment

Innovation

• Elastic Post-Training Sparsity
• Multi-Sparsity Optimization
• MS-HiLoRA
• Feature Mixer
• Model Compression