UniRank: Unified Rank Allocation for Low-Rank LLM Compression

2026-06-19

AI Summary

Existing low-rank decomposition methods either rely on handcrafted rank assignments with poor generalization or employ learning-based strategies that incur substantial computational overhead. This work proposes a unified rank allocation mechanism that formulates global low-rank compression as a ranking-and-truncation process, scoring singular components by integrating local singular energy ratios and global input-output cosine similarity—an attribute whose strong correlation with effective rank is revealed for the first time. Furthermore, the authors introduce a rank-preserving fine-tuning strategy that eliminates the need for re-merging, thereby avoiding information loss. The proposed method consistently achieves significant performance gains across diverse model architectures and compression settings, reducing perplexity by up to 50% over baselines without requiring fine-tuning in single-shot compression scenarios.

Abstract

Low-rank decomposition serves as a promising compression paradigm for large language models, however, rank allocation remains challenging: manual rules lack generalizability, and learning-based approaches incur heavy computational overhead. To address these issues, we formulate global low-rank allocation as a sorting-and-truncation pipeline, and score each singular component via dual criteria: \textbf{Local} singular energy ratio that quantifies the intrinsic importance within the decomposed parameter matrix and \textbf{Global} functional importance (measured by input-output cosine similarity) that evaluates the functional significance of decomposed modules. We verify the strong correlation between high input-output cosine similarity and low effective rank through geometric interpretation and experimental validation. Furthermore, we propose rank-preserving fine-tuning, which performs direct LoRA tuning on decomposed weights and avoids extra information loss caused by re-truncation in conventional merging pipelines. Empirical results confirm that our method delivers sustained performance enhancements when combined with models featuring distinct decomposition schemes, model sizes and architectural designs, e.g. in one-shot compression without further fine-tuning, our method reduces perplexity by up to 50\% compared with uniform and heuristic allocation baselines. Code will be available at https://github.com/EIT-NLP/LLM-Pruning.

Problem

• low-rank compression
• rank allocation
• large language models
• model compression
• singular value decomposition

Innovation

• low-rank decomposition
• rank allocation
• singular energy ratio
• input-output cosine similarity
• rank-preserving fine-tuning