model compression and sparsity

To address efficient deployment of large language and vision models on resource-constrained devices, this work systematically investigates the non-orthogonality between sparsification and quantization—two key model compression techniques—and characterizes their joint accuracy degradation mechanism. We provide the first rigorous mathematical proof of their non-orthogonality, revealing that error accumulation is intrinsic and critically dependent on operational ordering: quantizing before sparsifying severely degrades accuracy, whereas sparsifying before quantizing mitigates error propagation. Grounded in theoretical analysis and validated across diverse models (OPT, LLaMA-125M–8B, ViT, ResNet) and hardware platforms, we establish “sparsify-then-quantize” as the optimal practice. This principle achieves high compression ratios while markedly improving the accuracy–efficiency trade-off, offering both theoretically grounded insights and a practical, empirically verified paradigm for edge deployment of foundation models.

To address the low training efficiency and high computational/memory overhead of block-sparse models, this paper proposes an end-to-end differentiable training framework that abandons the conventional “dense-then-prune” paradigm, instead initializing directly from a sparse structure and dynamically optimizing block sizes during training. The method introduces three key components: gradient updates under structured sparsity constraints, block-level adaptive mask learning, and sparse-dense hybrid forward/backward propagation—ensuring both hardware compatibility and training stability. Experiments across multiple benchmarks demonstrate 40–65% reductions in computation and memory usage while matching the accuracy of dense baselines; moreover, the framework enables automatic block-size search. To our knowledge, this is the first structured sparse training approach that jointly optimizes block dimensions with model parameters during end-to-end training.

This study investigates how weight pruning, weight quantization, and activation quantization affect the pretraining scaling laws of large language models (LLMs), aiming to establish a unified effective-parameter scaling framework. It is the first to incorporate quantization into such a framework, theoretically modeling and empirically validating parameter efficiency across varying bit-widths and sparsity levels. Results show that weight quantization substantially improves parameter efficiency, though full quantization exhibits diminishing returns at ultra-low bit-widths. Through systematic ablation and cross-configuration modeling, we derive a unified scaling formula that predicts performance under diverse compression strategies. Key contributions are: (1) demonstrating that disparate compression techniques share a common effective-parameter scaling mechanism; (2) unifying quantization and pruning within a single theoretical framework; and (3) providing a composable, predictive foundation and optimization paradigm for efficient LLM design.

Conventional pruning methods suffer from severe accuracy collapse at high sparsity levels, failing to meet stringent hardware constraints on model size. To address this, we propose a bidirectional pruning-regeneration framework that departs from traditional unidirectional pruning: it first applies aggressive structured pruning, then dynamically restores critical connections based on importance estimation and performance feedback. This iterative co-optimization of pruning and selective connection regeneration effectively mitigates accuracy degradation under extreme compression. Experiments demonstrate that our method achieves an average accuracy improvement of 4.2% over state-of-the-art approaches at equivalent sparsity levels. Notably, on ResNet-50, it attains 95% sparsity while retaining over 98% of the original accuracy—substantially outperforming existing pruning techniques. The proposed framework establishes a new paradigm for deploying highly accurate, ultra-sparse models on resource-constrained edge devices.

To address the trade-off among model size, inference latency, and accuracy degradation when deploying deep neural networks on edge devices, this paper proposes two co-designed pruning-quantization joint optimization frameworks. Methodologically, it tightly integrates feature-map similarity–based filter pruning with adaptive power-of-two (APoT) quantization, jointly optimizing pruning masks and low-bit (≤4-bit) quantization parameters during training. The key contribution lies in leveraging the complementarity of pruning and APoT: pruning eliminates structural redundancy, while APoT enhances quantized representation efficiency—thereby avoiding error accumulation inherent in sequential compression. Experiments on ResNet and VGG demonstrate that our approach achieves 5.2× model size reduction, 6.8× FLOPs reduction, and 4.1× inference speedup, with ≤0.3% Top-1 accuracy drop relative to full-precision baselines—substantially outperforming standalone pruning or quantization methods and exhibiting strong practicality for edge deployment.

This work addresses the challenge of efficiently compressing large language models without training while minimizing performance degradation. The authors propose a training-free compression method that formulates layer-wise sparsity allocation as a multiple-choice knapsack problem, dynamically distributing sparsity across layers under a global compression budget using calibration data. Efficient weight reconstruction is achieved via a single-step sparse matrix factorization inspired by dictionary learning, circumventing iterative optimization and backpropagation to drastically reduce computational overhead. Experiments demonstrate superior performance over existing methods at compression rates of 20%–50%, with over 90% of the original model’s performance retained at 30% compression. Moreover, after light fine-tuning—e.g., compressing Qwen3-14B to 8B and refining with 30M tokens—the model nearly matches the performance of the native Qwen3-8B.

This work addresses the unclear impact of the execution order between pruning and quantization in joint compression on model performance. It presents the first systematic investigation into this ordering effect and proposes the “progressive intensity hypothesis,” which posits that weaker perturbations should precede stronger ones—a claim substantiated through theoretical perturbation analysis. Extensive experiments across large language and vision models, including complex scenarios such as multi-stage compression and mixed-precision quantization, validate the universality of this hypothesis. The results demonstrate that adhering to the progressive intensity ordering consistently yields significant performance improvements and exhibits strong generalization across diverse architectures and compression configurations.

This work addresses the discrepancy between conventional compression metrics—such as parameter count and FLOPs—and actual inference latency in CPU- and memory-constrained edge deployment scenarios, where such proxies often fail to reflect real-world performance. To bridge this gap, the authors propose a latency-driven, sequential compression pipeline that integrates unstructured pruning, INT8 quantization-aware training (QAT), and knowledge distillation (KD) within a unified training framework, jointly optimizing model accuracy, size, and inference speed. Experimental results demonstrate that this specific ordering significantly outperforms alternative combinations, achieving CPU inference latencies of 0.99–1.42 milliseconds on CIFAR-10/100 with ResNet-18, WRN-28-10, and VGG-16-BN models while maintaining high accuracy and compactness, thereby establishing a new paradigm for edge-oriented model compression under realistic latency constraints.

Efficient deployment of large language models (LLMs) in resource-constrained settings necessitates synergistic model compression, yet the optimal sequencing and interaction effects of knowledge distillation (KD), structured pruning, and low-bit quantization remain unclear. Method: We systematically investigate all permutations of these three techniques via controlled experiments on Qwen2.5-3B, evaluating their impact on both model performance and compression ratio. Contribution/Results: We identify pruning–KD–quantization (P-KD-Q) as the optimal cascade: structured pruning first preserves architectural redundancy; KD subsequently recovers accuracy lost during pruning; and quantization is applied last to avoid irreversible information loss from early low-bit approximation. This sequence achieves a 3.68× compression ratio while preserving strong language understanding and instruction-following capabilities. Our findings establish a reproducible, generalizable pipeline for LLM lightweighting—offering the first empirical evidence that compression order critically governs trade-offs between efficiency and fidelity.

Traditional three-stage Huffman coding incurs significant computational and latency overhead in multi-accelerator communication due to the need for real-time frequency analysis, codebook generation, and transmission, making it ill-suited for low-latency requirements. This work proposes a single-stage Huffman encoder that constructs a fixed codebook based on the average probability distribution derived from historical data batches, eliminating the need for real-time codebook generation and transmission. For the first time, this approach is applied to lossless compression of cross-layer and sharded tensors. Evaluated on the Gemma 2B model, the method achieves a compression ratio only 0.5% lower than per-shard Huffman coding and lies within 1% of the Shannon limit, substantially improving communication efficiency while closely approaching the theoretical compression bound.