Understanding Calibration and Truncation Error Propagation in Training-Free Low-Rank Compression for LLMs

2026-08-09

AI Summary

This work addresses a critical inconsistency in existing training-free low-rank compression methods, where layer-wise accumulation of activation residuals during calibration and distortion of post-compression layer importance distributions lead to a mismatch between compressed models and their actual inference behavior. The paper presents the first systematic analysis of how calibration and truncation errors propagate across layers and introduces a novel two-stage, training-free correction strategy. By integrating per-layer calibration correction with dynamic rank allocation during iterative compression, the method effectively mitigates error propagation and layer importance misalignment. Compatible with mainstream low-rank decomposition frameworks, it achieves zero-shot accuracy gains of 1–2.5 percentage points on Llama and Qwen3 models, substantially improving compression fidelity.

Abstract

Training-free low-rank compression frameworks have been gaining prominence for LLM compression given their effectiveness in reducing model parameter count while maintaining task-level accuracy. However, existing SOTA frameworks share two key limitations: (1) residual errors in calibration data activations accumulate across layers during compression, causing misalignment between representations simulated at compression time and those experienced at inference; (2) the assumption that layer importance distribution is preserved post-compression does not hold. Together, these two effects introduce misalignment in the compression process in relation to the deployed model. We study these effects and propose a simple, training-free methodology compatible with existing frameworks to mitigate them, comprising: (1) Layer-by-Layer Compression with Calibration Correction; (2) Iterative Compression with Rank Allocation Correction. Implemented atop an existing SOTA decomposition framework, and evaluated on Llama and Qwen3 models across various benchmarks and compression rates, our approach demonstrates up to ~1-2.5 accuracy point improvements over per-weight and joint decomposition baselines on zero-shot tasks.

Problem

• calibration error
• truncation error
• low-rank compression
• LLM compression
• error propagation

Innovation

• training-free compression
• calibration error propagation
• rank allocation
• layer-wise compression
• low-rank decomposition