TORQ: Two-Level Orthogonal Rotation for MXFP4 Quantization

2026-05-19

AI Summary

This work addresses the significant accuracy degradation in MXFP4 activation quantization, which stems from a structural mismatch between activation distributions and the block floating-point format. The study reveals, for the first time, that this mismatch manifests as two interrelated issues: inter-block energy imbalance and low intra-block codebook utilization. To mitigate these problems, the authors propose a training-free post-training quantization framework that employs a two-level orthogonal rotation strategy. At the macro level, inter-block rotations based on the Schur-Horn theorem balance activation variances across blocks; at the micro level, intra-block rotations guided by maximum entropy principles enhance codebook efficiency. Evaluated on Qwen3-32B, the method reduces WikiText perplexity to 8.43 and improves average accuracy from 38.40% to 73.63%, substantially narrowing the performance gap between 4-bit floating-point quantized models and their full-precision counterparts.

Abstract

As Large Language Models (LLMs) advance toward practical deployment, the Microscaling FP4 (MXFP4) format has emerged as a cornerstone for next-generation low-bit inference, owing to its ability to balance high dynamic range with hardware efficiency. However, directly applying MXFP4 to LLM activation quantization inevitably leads to significant accuracy degradation. In this paper, we theoretically analyze the error structure of MXFP4 activation quantization, revealing that the root cause of this performance drop lies in two structural imbalances between activation distributions and the MXFP4 block floating-point format: (1) extreme inter-block variance imbalance and (2) intra-block codebook utilization imbalance. To address these challenges, we propose TORQ (Two-level Orthogonal Rotation for MXFP4 Quantization), a training-free Post-Training Quantization (PTQ) framework designed to reshape the geometric properties of the activation space through optimal coordinate transformations. At the macroscopic level, TORQ leverages the Schur-Horn theorem to redistribute activation energy via inter-block orthogonal rotation, preventing high-variance blocks from driving up shared scaling factors and thereby preserving the precision of small-magnitude elements. At the microscopic level, TORQ employs maximum-entropy-guided intra-block rotation to alleviate codebook collapse and maximize the MXFP4 codebook's information capacity. Experiments on mainstream LLMs such as LLaMA3 and Qwen3 show that TORQ significantly improves the accuracy of MXFP4 activation quantization compared to existing methods: on Qwen3-32B, the perplexity on WikiText is reduced to 8.43 (vs. 7.61 for BF16), and the average accuracy increases from 38.40% with direct RTN to 73.63% (vs. 74.82% for BF16), substantially narrowing the gap between 4-bit floating-point quantization and full-precision inference.

Problem

• MXFP4 quantization
• activation quantization
• accuracy degradation
• structural imbalance
• large language models

Innovation

• MXFP4 quantization
• orthogonal rotation
• post-training quantization
• activation quantization
• codebook utilization