🤖 AI Summary
Efficient deployment of large language models (LLMs) in resource-constrained settings necessitates synergistic model compression, yet the optimal sequencing and interaction effects of knowledge distillation (KD), structured pruning, and low-bit quantization remain unclear.
Method: We systematically investigate all permutations of these three techniques via controlled experiments on Qwen2.5-3B, evaluating their impact on both model performance and compression ratio.
Contribution/Results: We identify pruning–KD–quantization (P-KD-Q) as the optimal cascade: structured pruning first preserves architectural redundancy; KD subsequently recovers accuracy lost during pruning; and quantization is applied last to avoid irreversible information loss from early low-bit approximation. This sequence achieves a 3.68× compression ratio while preserving strong language understanding and instruction-following capabilities. Our findings establish a reproducible, generalizable pipeline for LLM lightweighting—offering the first empirical evidence that compression order critically governs trade-offs between efficiency and fidelity.
📝 Abstract
Large Language Models (LLMs) require substantial computational resources, making model compression essential for efficient deployment in constrained environments. Among the dominant compression techniques: knowledge distillation, structured pruning, and low-bit quantization, their individual effects are well studied, but their interactions and optimal sequencing remain unclear. This work systematically examines how these techniques perform both independently and in combination when applied to the Qwen2.5 3B model. We evaluate multiple compression pipelines, including single, and proposed three-technique sequences, using perplexity, G-Eval, clarity, prompt alignment, and compression ratio as metrics. Our experiments show that quantization provides the greatest standalone compression, while pruning introduces moderate quality degradation. Critically, the ordering of techniques significantly affects the final model quality: the sequence Pruning, Knowledge Distillation, Quantization (P-KD-Q) yields the best balance, achieving a 3.68x compression ratio while preserving strong instruction-following and language understanding capabilities. Conversely, pipelines applying quantization early suffer severe performance degradation due to irreversible information loss that impairs subsequent training. Overall, this study offers practical insight into designing effective, ordering-aware compression pipelines for deploying LLMs in resource-limited settings.