MoDeGPT: Modular Decomposition for Large Language Model Compression

2024-08-19

arXiv.org

AI Summary

Deploying large language models (LLMs) on resource-constrained devices faces bottlenecks including high computational overhead, accuracy degradation after compression, and reliance on post-compression fine-tuning. Method: We propose MoDeGPT, a fine-tuning-free modular structured compression framework that decouples Transformer blocks into matrix-pair modules and applies unified Nyström, Column-Randomization (CR), and SVD decompositions at the module level to reconstruct outputs and reduce hidden dimensionality. This introduces the first gradient-free, module-level decomposition paradigm, theoretically unifying three major matrix approximation techniques. Contribution/Results: MoDeGPT achieves synergistic optimization of high compression ratio and zero-shot performance: 98% reduction in FLOPs for a 13B model; 90–95% zero-shot accuracy retention for Llama-2/3 and OPT under 25–30% parameter compression; compression completed in hours on a single GPU; and up to 46% inference throughput improvement.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models (LLMs) have reshaped the landscape of artificial intelligence by demonstrating exceptional performance across various tasks. However, substantial computational requirements make their deployment challenging on devices with limited resources. Recently, compression methods using low-rank matrix techniques have shown promise, yet these often lead to degraded accuracy or introduce significant overhead in parameters and inference latency. This paper introduces extbf{Mo}dular extbf{De}composition (MoDeGPT), a novel structured compression framework that does not need recovery fine-tuning while resolving the above drawbacks. MoDeGPT partitions the Transformer block into modules comprised of matrix pairs and reduces the hidden dimensions via reconstructing the module-level outputs. MoDeGPT is developed based on a theoretical framework that utilizes three well-established matrix decomposition algorithms -- Nystr""om approximation, CR decomposition, and SVD -- and applies them to our redefined transformer modules. Our comprehensive experiments show MoDeGPT, without backward propagation, matches or surpasses previous structured compression methods that rely on gradient information, and saves 98% of compute costs on compressing a 13B model. On extsc{Llama}-2/3 and OPT models, MoDeGPT maintains 90-95% zero-shot performance with 25-30% compression rates. Moreover, the compression can be done on a single GPU within a few hours and increases the inference throughput by up to 46%.

Problem

• Reduces computational requirements for deploying large language models on resource-limited devices.
• Addresses accuracy degradation and overhead in parameters and inference latency from compression methods.
• Introduces a structured compression framework without needing recovery fine-tuning.

Innovation

• Modular decomposition for LLM compression
• Uses Nyström, CR, and SVD matrix decompositions
• Achieves high compression with minimal accuracy loss