Activation- and Influence-Aware Ranks (AIR): Function-Preserving SVD Compression for LLMs

2026-06-18

AI Summary

This work addresses the challenge of simultaneously achieving substantial parameter reduction and performance preservation in large language model (LLM) compression by proposing an activation- and influence-aware low-rank approximation method. The approach uniquely integrates element-wise backward influence metrics into singular value decomposition (SVD)-based compression and employs a single closed-form alternating least squares (ALS) step to preserve functionality, offering both locality and monotonic descent properties. It is also orthogonal to end-to-end fine-tuning techniques. Experimental results demonstrate that with ≤60% of the original parameters retained, the method improves perplexity by over 18% compared to SVD-LLM(W). Moreover, it achieves comparable model quality using only ~10% calibration data while significantly reducing FLOPs, peak memory usage, and per-token latency.

Abstract

We present Activation- and Influence-Aware Ranks (AIR), an SVD-based LLM compression framework that guides each weight matrix's low-rank approximation with a backward-signal influence metric. Starting from the activation-aware optimum of SVD-LLM(W), AIR runs a single closed-form alternating least squares (ALS) sweep that integrates influence element-wise under a monotone-descent guarantee. AIR is layer-local and composes orthogonally with end-to-end methods: alone it exceeds ACIP, and AIR+LoRA outperforms it further. AIR improves perplexity over SVD-LLM(W) by >18% at <=60% parameter retention, matches its quality with ~90% less calibration data, and turns parameter savings into FLOP, peak-memory, and per-token latency gains.

Problem

• LLM compression
• low-rank approximation
• function preservation
• SVD
• parameter efficiency

Innovation

• low-rank approximation
• SVD compression
• influence-aware
• activation-aware
• alternating least squares