IO-SVD: Input-Output Whitened SVD for Adaptive-Rank LLM Compression

2026-05-15

AI Summary

This work addresses the high storage and computational costs of large language models in resource-constrained settings, where existing SVD-based compression methods struggle to preserve performance under aggressive compression. The authors propose IO-SVD, a post-training low-rank compression technique that constructs an input–output bilateral whitening space and introduces a KL divergence–based predictive sensitivity metric combined with activation statistics to enable loss-aware, heterogeneous rank allocation. Innovatively integrating output-side whitening, second-order KL loss expansion, first-order gradient calibration, and adaptive rank pruning—alongside 8-bit quantization remapping—IO-SVD achieves minimal performance degradation at high compression ratios across diverse large language and vision-language models, while significantly accelerating practical inference speed.

Abstract

Large language models deliver strong performance across language and reasoning tasks, but their storage and compute costs remain major barriers to deployment in resource-constrained and latency-sensitive settings. SVD-based post-training compression offers a hardware-agnostic way to reduce model size and improve inference efficiency through low-rank factorization. However, existing methods often rely on input-only whitening spaces, homogeneous rank allocation, or loss-agnostic allocation heuristics, limiting their ability to preserve model quality under aggressive compression. We propose Input-Output Whitened SVD (IO-SVD), a post-training compression method that forms a KL-aware double-sided whitening space for model weights. Using a second-order expansion of the KL loss over the top-K token probabilities, IO-SVD constructs an output-side metric that captures predictive sensitivity, while input whitening captures activation statistics. We further introduce an efficient heterogeneous rank-allocation strategy that scores whitened singular components using first-order calibration loss estimates and prunes the least sensitive components under a global budget. Inspired by prior work that combines SVD truncation with quantization, we improve hybrid SVD-quantization compression through loss-aware remapping, which selects low-rank factor rows for 8-bit quantization based on the predicted loss change incurred by quantizing them. Extensive experiments across diverse LLM and VLM families, and inference-time analysis shows that IO-SVD compresses LLMs with minimal performance degradation while delivering practical inference speedups. Code is available at https://github.com/mint-vu/IO-SVD.git

Problem

• LLM compression
• SVD
• post-training compression
• rank allocation
• model efficiency

Innovation

• IO-SVD
• input-output whitening
• heterogeneous rank allocation
• KL-aware compression
• loss-aware quantization