ProcrustesGPT: Compressing LLMs with Structured Matrices and Orthogonal Transformations

2025-06-03

AI Summary

To address the low approximation accuracy and reliance on fine-tuning in large language model (LLM) compression via structured matrix approximations, this paper proposes an orthogonal pre-transformation mechanism: leveraging the output invariance of weight matrices under orthogonal transformations, it introduces learnable orthogonal matrices to enhance the fidelity of low-rank, circulant, or Toeplitz structural representations. We are the first to incorporate Procrustes analysis into LLM compression, designing an efficient solver based on singular value decomposition (SVD) and Householder reflections. The method enables fine-tuning-free compression without altering original weights. Evaluated on LLaMA-2 and Phi-3, it achieves 2–4× parameter reduction and 1.8× inference speedup, with perplexity degradation under 0.5—significantly outperforming existing fine-tuning-free approaches.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large language models (LLMs) demonstrate impressive results in natural language processing tasks but require a significant amount of computational and memory resources. Structured matrix representations are a promising way for reducing the number of parameters of these models. However, it seems unrealistic to expect that weight matrices of pretrained models can be accurately represented by structured matrices without any fine-tuning. To overcome this issue, we utilize the fact that LLM output is invariant under certain orthogonal transformations of weight matrices. This insight can be leveraged to identify transformations that significantly improve the compressibility of weights within structured classes. The proposed approach is applicable to various types of structured matrices that support efficient projection operations. Code is available at https://github.com/GrishKate/ProcrustesGPT

Problem

• Reducing computational and memory resources in LLMs
• Improving compressibility of pretrained model weights
• Utilizing orthogonal transformations for structured matrices

Innovation

• Uses structured matrices for LLM compression
• Applies orthogonal transformations for weight optimization
• Supports efficient projection operations