Problem
Research questions and friction points this paper is trying to address.
MoE Compression
Post-Compression Adjustment
Performance Recovery
Beyond Retraining-Free MoE Compression: A Cost-Normalized Study of Post-Compression Adjustment
🤖 AI Summary
研究通过小规模后压缩调整阶段改进无重训练MoE压缩，使用LM微调和教师KD方法，在成本控制下恢复压缩损失的性能。
📝 Abstract
Retraining-free MoE compression reduces deployment memory by pruning or merging experts, but often treats the compressed checkpoint as the final artifact. We argue that this view is incomplete: compressed MoE checkpoints are better understood as compressed initializations that benefit from a tiny post-compression adjustment stage. Across two MoE LLM backbones, four pruning/merging methods, three expert-retention ratios, and 28 benchmarks, we compare LM fine-tuning and teacher-based KD under matched small-data budgets and measured GPU costs. Using only 3,000 C4 examples and a single epoch of adjustment, Full FT recovers 37.3% of the original-to-compressed performance gap on average. Moreover, LM fine-tuning is more cost-effective than standard token-level KD, and full-parameter adjustment gives the strongest cost--recovery trade-off among the tested scopes. These results suggest that retraining-free compression should be paired with small post-compression adjustment to recover a substantial portion of the performance lost during compression.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Retraining-free MoE compression
Post-compression adjustment
Cost-effective fine-tuning
🔎 Similar Papers
No similar papers found.
