Optimizing Singular Spectrum for Large Language Model Compression

2025-02-20

AI Summary

Large language models (LLMs) face deployment challenges due to their massive parameter counts; existing singular value decomposition (SVD)-based compression methods naively rank component importance by singular value magnitude, ignoring task-specific relevance. Method: We propose SoCo, a novel framework featuring a learnable singular spectrum rescaling mechanism that dynamically assesses task-aware importance of SVD components in a data-driven manner. SoCo employs a three-stage progressive training strategy: coarse-grained compression, fine-grained sparsification, and importance-amplified compensation—enabling adaptive performance recovery under high compression ratios. Results: Evaluated across multiple LLMs and benchmark tasks, SoCo achieves an average 3.2% improvement in downstream accuracy at equivalent compression ratios, significantly outperforming state-of-the-art methods.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Data Mining & Knowledge Management: Data Compression

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities, yet prohibitive parameter complexity often hinders their deployment. Existing singular value decomposition (SVD) based compression methods simply deem singular values as importance scores of decomposed components. However, this importance ordered by singular values does not necessarily correlate with the performance of a downstream task. In this work, we introduce SoCo (Singular spectrum optimization for large language model Compression), a novel compression framework that learns to rescale the decomposed components of SVD in a data-driven manner. Concretely, we employ a learnable diagonal matrix to assign importance scores for singular spectrum and develop a three-stage training process that progressively refines these scores from initial coarse compression to fine-grained sparsification-thereby striking an effective balance between aggressive model compression and performance preservation. Thanks to the learnable singular spectrum, SoCo adaptively prunes components according to the sparsified importance scores, rather than relying on the fixed order of singular values. More importantly, the remaining components with amplified importance scores can compensate for the loss of the pruned ones. Experimental evaluations across multiple LLMs and benchmarks demonstrate that SoCo surpasses the state-of-the-art methods in model compression.

Problem

• Optimizing singular spectrum for compression
• Data-driven rescaling of SVD components
• Balancing compression and performance preservation

Innovation

• Learns to rescale SVD components
• Uses learnable diagonal matrix
• Three-stage training process