Mind the Approximation: Fisher-Weighted SVD Compression for ViTs

2026-09-07

AI Summary

研究解决了Vision Transformers压缩时精度损失问题，提出FACTS方法，通过改进Fisher加权SVD和引入快速约束秩搜索来优化模型压缩效果。

Abstract

Model compression is key to mitigate deployment challenges of ever growing machine learning models. In this area of research, singular value decomposition (SVD)-based compression offers a compelling trade-off between computational efficiency and model accuracy. Fisher-weighted SVD in particular provides principled, loss-aware compression. However, we find that improving the fidelity of Fisher approximation used in the compression is poorly predictive of post-compression accuracy for Vision Transformers (ViTs). Motivated by this observation, we propose FACTS, a structured Fisher Approximation tailored to Compressing ViTs with Fisher-weighted SVD, which enforces token-local aggregation while preserving within-token activation-gradient dependence. Additionally, we introduce a fast Constrained Rank Search (CoRS), that optimizes layer-wise rank allocation while adhering to a fixed floating point operation (FLOP) constraint. Extensive experiments across ViTs and hybrid architectures demonstrate that FACTS consistently improves accuracy-efficiency trade-offs without requiring finetuning. Notably, it outperforms the strongest SVD baseline by up to +5.8 percentage points (p.p.) Top-1 on Swin-B, with further gains driven by our search method. Code is available at https://github.com/MoritzTho/FACTS.

Problem

• Fisher-Weighted SVD
• Compression
• Vision Transformers
• Model Accuracy
• Fidelity

Innovation

• Fisher-Weighted SVD
• Vision Transformers
• Structured Fisher Approximation
• Constrained Rank Search
• Compression