FLAT-LLM: Fine-grained Low-rank Activation Space Transformation for Large Language Model Compression

2025-05-29

AI Summary

To address the high computational and memory overhead of deploying large language models (LLMs) in resource-constrained settings, this paper proposes a training-free, fine-grained low-rank compression method operating in the activation space. Specifically, it applies principal component analysis (PCA) per attention head, truncates singular vectors to perform weight transformation, and introduces an importance-driven dynamic rank allocation mechanism to adaptively optimize rank configurations across decoder layers. Unlike existing low-rank decomposition approaches, our method requires no fine-tuning (only minutes for calibration), preserves the original model architecture, substantially accelerates inference, and incurs negligible accuracy degradation. Extensive evaluation across four mainstream LLMs and eleven downstream tasks demonstrates consistent superiority over structural pruning baselines, validating both its efficiency and strong generalization capability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Large Language Models (LLMs) have enabled remarkable progress in natural language processing, yet their high computational and memory demands pose challenges for deployment in resource-constrained environments. Although recent low-rank decomposition methods offer a promising path for structural compression, they often suffer from accuracy degradation, expensive calibration procedures, and result in inefficient model architectures that hinder real-world inference speedups. In this paper, we propose FLAT-LLM, a fast and accurate, training-free structural compression method based on fine-grained low-rank transformations in the activation space. Specifically, we reduce the hidden dimension by transforming the weights using truncated eigenvectors computed via head-wise Principal Component Analysis (PCA), and employ an importance-based metric to adaptively allocate ranks across decoders. FLAT-LLM achieves efficient and effective weight compression without recovery fine-tuning, which could complete the calibration within a few minutes. Evaluated across 4 models and 11 datasets, FLAT-LLM outperforms structural pruning baselines in generalization and downstream performance, while delivering inference speedups over decomposition-based methods.

Problem

• High computational demands of LLMs limit deployment in resource-constrained environments
• Existing low-rank compression methods cause accuracy loss and slow inference
• Need for efficient, training-free compression without performance degradation

Innovation

• Fine-grained low-rank activation space transformation
• Head-wise Principal Component Analysis (PCA)
• Importance-based adaptive rank allocation