Different Prompts, Different Ranks: Prompt-aware Dynamic Rank Selection for SVD-based LLM Compression

2026-05-08

AI Summary

This work addresses the limitations of existing static SVD-based compression methods, which employ a fixed rank and thus struggle to adapt to the varying optimal rank requirements across different prompts, while also relying heavily on calibration data. To overcome these issues, the authors propose PARSE, a novel framework that enables prompt-aware dynamic rank selection for the first time. PARSE employs a lightweight linear routing module trained offline to predict the optimal rank for each input prompt and leverages the observation that semantically similar prompts exhibit stable rank-selection patterns, introducing a pattern caching mechanism to accelerate inference. By decoupling rank selection from calibration set dependency and integrating expert memory aggregation with kernel fusion techniques, PARSE achieves up to a 10% average accuracy improvement at a 0.6 compression ratio on LLaMA-7B, along with up to 2.5× faster prefill and 2.4× faster decode speeds compared to static SVD.

Abstract

Large language models (LLMs) have rapidly grown in scale, creating substantial memory and computational costs that hinder efficient deployment. Singular value decomposition (SVD) has emerged as an effective post-training compression technique, but existing SVD-based methods rely on static rank truncation, applying a fixed prefix of singular components to all inputs regardless of their diversity. We identify two limitations of this static design: the optimal rank varies across individual prompts, and the selected rank is sensitive to the choice of calibration set, leading to suboptimal performance across diverse inputs. To address these challenges, we propose $\textbf{PARSE}$, a post-training framework for $\textbf{P}$rompt-$\textbf{A}$ware $\textbf{R}$ank $\textbf{S}$election as $\textbf{E}$xperts in SVD-compressed LLMs. PARSE trains a linear router offline to perform prompt-aware rank selection, decoupling it from calibration information by supervising the router against dense-model outputs on a large-scale corpus. We further observe that rank-selection patterns are shared across semantically similar prompts and remain stable across decoding steps, allowing appropriate rank subsets to be served directly from a pattern cache at inference. Complemented by expert memory aggregation and kernel fusion for system-level efficiency, PARSE is orthogonal to existing SVD-based pipelines and consistently improves both model quality and inference efficiency. Integrated with four representative SVD-based methods, PARSE improves average task accuracy by up to 10% at a compression ratio of 0.6 on LLaMA-7B, and achieves up to 2.5 $\times$ prefill and 2.4 $\times$ decode speedup over native SVD execution.

Problem

• SVD-based compression
• static rank truncation
• prompt diversity
• calibration set sensitivity
• LLM compression

Innovation

• Prompt-aware
• Dynamic Rank Selection
• SVD Compression
• LLM Efficiency
• Linear Router