LACE-SVD: Loss-Aware SVD with Cumulative Error Correction for LLM Compression

2026-07-03

AI Summary

Existing SVD-based compression methods overlook inter-layer loss sensitivity and the cumulative propagation of local errors through residual streams, leading to significant performance degradation at high compression ratios. This work proposes a loss-aware SVD compression framework that, for the first time, incorporates layer-wise loss sensitivity into rank budget allocation by dynamically assigning ranks based on changes in negative log-likelihood. Furthermore, it introduces a closed-form local update rule coupled with a propagation-aware residual stream correction mechanism to explicitly model and suppress error accumulation. Evaluated on LLaMA-7B at a 0.6 compression ratio, the method achieves a WikiText-2 perplexity of 32.57, substantially outperforming Dobi-SVD (46.18), thereby demonstrating its effectiveness and superiority.

Abstract

The rapid growth in the parameter scale of large language models (LLMs) has created a strong demand for efficient compression techniques. As a hardware-agnostic and highly compatible approach, low-rank compression has been widely adopted to reduce both memory footprint and computational cost. However, existing SVD-based methods are still largely driven by local reconstruction objectives, overlooking two critical limitations: rank budgets are often allocated without explicitly considering layer-wise loss sensitivity, and local approximation errors can propagate and accumulate through the residual stream, leading to amplified global deviations from the original model. To address these issues, we propose LACE-SVD, a Loss-Aware SVD framework with Cumulative Error correction for LLM compression. LACE-SVD first estimates the calibration negative-log-likelihood increase induced by candidate layer-wise compression ratios and solves a budget-constrained allocation problem to assign rank budgets. It then refines the compressed model with closed-form local updates and introduces a propagation-aware correction for residual-stream output modules, reducing layer-output discrepancy as a proxy for cumulative error propagation. Experimental results demonstrate that at a high compression ratio (0.6), the WikiText-2 PPL of our method on LLaMA-7B (32.57) is significantly better than that of Dobi-SVD (46.18).

Problem

• LLM compression
• SVD
• rank allocation
• error propagation
• loss sensitivity

Innovation

• Loss-Aware Compression
• Cumulative Error Correction
• Low-Rank Approximation
• SVD-based Model Compression
• Residual Stream Propagation