LowRankArena: A Standardized Evaluation Platform for SVD-Based LLM Compression

2026-08-26

AI Summary

为解决SVD压缩LLM时缺乏统一评估标准的问题，提出LowRankArena平台，通过标准化任务版本、压缩预算等，实现可重复的评估。

Abstract

SVD-based low-rank compression has become a fast-growing direction for reducing the memory and computational cost of large language models (LLMs). However, meaningful comparison across existing studies remains difficult as prior evaluations use varied benchmarks, inconsistent ratios, and diverse setups, often failing to isolate low-rank effects from auxiliary techniques. As a result, it remains unclear whether reported gains reflect method-level improvements or differences in evaluation protocol. This lack of comparability highlights the need for a unified, reproducible evaluation platform. To address this problem, we present LowRankArena, a standardized evaluation platform for SVD-based LLM compression. LowRankArena unifies task versions, uniform-precision compression budgets, comparison regimes, and inference measurements, and provides a reproducible pipeline with over 3 TiB released compressed checkpoints. Using LowRankArena, our aligned audit of five representative SVD methods reveals that prior findings are highly conditional under standardized protocols: clear leaders and performance tiers shift across backbones and keep ratios, multiple-choice accuracy can hide large perplexity degradation, and nominal low-rank savings yield workload-dependent and often limited end-to-end speedups. Our code is available at: https://github.com/Zishan-Shao/lowrankarena.git.

Problem

• SVD-based compression
• evaluation platform
• low-rank compression
• large language models
• benchmark

Innovation

• SVD-based LLM compression
• Standardized Evaluation Platform
• Reproducible Pipeline
• Unified Task Versions
• Uniform-Precision Compression Budgets