ReSpinQuant: Efficient Layer-Wise LLM Quantization via Subspace Residual Rotation Approximation

2026-04-13

AI Summary

This work addresses the significant accuracy degradation in large language model quantization caused by activation outliers. While existing global rotation methods lack sufficient expressiveness and layer-wise rotation introduces substantial inference overhead, the proposed ReSpinQuant framework achieves the first offline fusion of layer-wise activation rotation. By approximating residual subspace rotations and integrating them into fused rotation matrices, ReSpinQuant preserves the high representational capacity of adaptive per-layer transformations while incurring negligible inference latency. Combined with post-training quantization, the method attains state-of-the-art accuracy under W4A4 and W3A3 settings, substantially outperforming global rotation approaches with almost no additional computational cost at inference time.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Large Vision Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Measurement, analysis, and circumvention of Web censorship
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Rotation-based Post-Training Quantization (PTQ) has emerged as a promising solution for mitigating activation outliers in the quantization of Large Language Models (LLMs). Global rotation methods achieve inference efficiency by fusing activation rotations into attention and FFN blocks, but suffer from limited expressivity as they are constrained to use a single learnable rotation matrix across all layers. To tackle this, layer-wise transformation methods emerged, achieving superior accuracy through localized adaptation. However, layer-wise methods cannot fuse activation rotation matrices into weights, requiring online computations and causing significant overhead. In this paper, we propose ReSpinQuant, a quantization framework that resolves such overhead by leveraging offline activation rotation fusion and matching basis using efficient residual subspace rotation. This design reconciles the high expressivity of layer-wise adaptation with only negligible inference overhead. Extensive experiments on W4A4 and W3A3 quantization demonstrate that ReSpinQuant achieves state-of-the-art performance, outperforming global rotation methods and matching the accuracy of computationally expensive layer-wise methods with minimal overhead.

Problem

• Large Language Models
• Post-Training Quantization
• Layer-Wise Quantization
• Activation Outliers
• Inference Overhead

Innovation

• layer-wise quantization
• rotation-based PTQ
• residual subspace rotation
• activation outlier mitigation
• offline rotation fusion