Residual Sparsification via Output Importance for Compressing Mixture-of-Experts LLMs

2026-08-31

AI Summary

为解决MoE架构的大语言模型GPU内存需求大的问题，提出PARSER方法，通过保留输出误差来压缩残差矩阵，提高模型精度。

Abstract

Mixture-of-experts (MoE) architectures scale large language models efficiently, but they demand massive GPU memory. To cope with such demand, models are commonly compressed to reduce their memory footprint. Residual sparsification is a representative compression technique that decomposes each projection matrix of an expert into a shared base matrix and per-expert residual matrix, and then compresses the residuals. Existing sparsification methods compress each residual matrix independently by minimizing its compression error, thereby minimizing the error of each projection matrix. However, our analysis shows that this objective is misaligned with preserving model accuracy after compression. In an expert, the final output is produced through computations coupled across multiple projections and hidden representations. Therefore, even small errors in individual matrices can propagate through hidden representations and projection interactions, leading to large expert output errors and accuracy degradation. To address this misalignment, we propose PARSER, a new residual sparsification method that shifts the compression objective from minimizing isolated matrix errors to preserving the expert output error. PARSER achieves this by introducing output importance, which measures the actual contribution to the expert output error. Our experiments show that, compared with existing methods, PARSER narrows the accuracy gap to the uncompressed model by 1.41$\times$ on Qwen and 1.44$\times$ on DeepSeek, while achieving the same peak memory reduction.

Problem

• Residual Sparsification
• Mixture-of-Experts
• Model Compression
• Output Error

Innovation

• Residual Sparsification
• Output Importance
• Mixture-of-Experts
• Compression
• Model Accuracy