SVD-Surgeon: Optimal Singular-Value Surgery for Large Language Model Compression

2026-06-22

AI Summary

This work addresses the challenge of deploying large language models due to their high computational and memory demands. While existing low-rank compression methods based on singular value decomposition (SVD) suffer from significant performance degradation caused by truncation-induced errors and lack effective loss compensation mechanisms, this paper introduces the Optimal Brain Surgeon (OBS) framework into SVD-based model compression for the first time. By leveraging a second-order Taylor expansion, the method derives a closed-form optimization for retained singular values and establishes a pruning criterion grounded in parameter saliency, thereby achieving second-order compensation for truncation error. Notably, the approach requires no retraining and consistently outperforms current SVD compression techniques on OPT and LLaMA-2-7B models, yielding substantially lower perplexity at identical compression ratios.

Abstract

Large language models (LLMs) achieve remarkable performance across a wide range of tasks, but their deployment is constrained by substantial memory and compute requirements. Low-rank compression via singular value decomposition (SVD) is an effective remedy, but existing methods focus on how to factorize and which components to keep. We introduce SVD-Surgeon, a training-free method that brings the Optimal Brain Surgeon (OBS) framework to the singular-value basis. Treating each singular value as a parameter, it computes a closed-form update of the retained singular values that compensates, to second order in the model loss, for those removed by truncation. The same analysis yields a saliency for choosing which values to prune. As it operates directly on the singular-value factorization, SVD-Surgeon can be layered on top of existing SVD compressors. Applied to SVD-LLM, a leading SVD-based method, it improves the perplexity-compression trade-off on the OPT family and LLaMA 2-7B without any retraining.

Problem

• large language models
• model compression
• singular value decomposition
• memory efficiency
• compute requirements

Innovation

• SVD-Surgeon
• Optimal Brain Surgeon
• singular value decomposition
• model compression
• training-free pruning