All for 1-Bit: Towards Genuine 1-Bit Post-Training Quantization for LLMs
为解决大型语言模型部署效率问题，提出All for 1-Bit框架，通过二值化因子分解和层次化夏普利分配方法实现真正的1比特后训练量化。
0 citationsRead paper
Selected work
Representative Papers
TWLA: Achieving Ternary Weights and Low-Bit Activations for LLMs via Post-Training Quantization
This work addresses the challenge of efficiently deploying large language models, which are hindered by high memory and computational demands, while existing ternarization methods struggle to achieve end-to-end low-bit inference due to heavy-tailed activation distributions. To overcome this, the authors propose TWLA, a post-training quantization framework that enables significantly accelerated inference with high accuracy using only 1.58-bit weights and 4-bit activations (W1.58A4). The approach introduces three key innovations: an Euclidean-to-manifold asymmetric ternary quantizer (E2M-ATQ), Kronecker orthogonal trimodal shaping (KOTMS), and inter-layer aware mixed-precision activation allocation (ILA-AMP). Together, these components enable joint optimization of weight ternarization and activation quantization, achieving high-fidelity low-bit representations in manifold space.
0 citationsRead paper
MGVQ: Synergizing Multi-dimensional Sensitivity-Aware and Gradient-Hessian Fusion for Vector Quantization
Existing vector quantization methods for vision-language models struggle to account for cross-modal weight distribution discrepancies and introduce compensation bias due to the neglect of first-order gradient information. To address these limitations, this work proposes MGVQ, a novel framework that jointly models channel sensitivity and gradient–Hessian information for highly efficient ultra-low-bit quantization. The core innovations include a mixed-precision allocation strategy guided by multidimensional sensitivity analysis and a second-order error compensation mechanism integrating gradient embedding with Kronecker and Block-LDL decompositions. Experimental results demonstrate that MGVQ substantially improves performance across multiple state-of-the-art vision-language models—LLaVA-OneVision, InternVL2, and Qwen2-VL—with gains of up to 4.9 accuracy points at 2-bit quantization, achieving 71.4% on InternVL2-26B.
0 citationsRead paper
TORQ: Two-Level Orthogonal Rotation for MXFP4 Quantization
This work addresses the significant accuracy degradation in MXFP4 activation quantization, which stems from a structural mismatch between activation distributions and the block floating-point format. The study reveals, for the first time, that this mismatch manifests as two interrelated issues: inter-block energy imbalance and low intra-block codebook utilization. To mitigate these problems, the authors propose a training-free post-training quantization framework that employs a two-level orthogonal rotation strategy. At the macro level, inter-block rotations based on the Schur-Horn theorem balance activation variances across blocks; at the micro level, intra-block rotations guided by maximum entropy principles enhance codebook efficiency. Evaluated on Qwen3-32B, the method reduces WikiText perplexity to 8.43 and improves average accuracy from 38.40% to 73.63%, substantially narrowing the performance gap between 4-bit floating-point quantized models and their full-precision counterparts.
0 citationsRead paper
CAR-SAM: Cross-Attention Reconstruction for Post-Training Quantization of the Segment Anything Model
Existing post-training quantization methods for the Segment Anything Model (SAM) overlook the cross-attention structure in its decoder, leading to attention information dispersion and unstable reconstruction optimization, which severely degrades segmentation performance at low bit-widths. To address this, this work proposes CAR-SAM, the first quantization framework specifically tailored to SAM’s cross-attention architecture. It introduces a MatMul-aware compensation mechanism to mitigate attention dispersion and a joint cross-attention reconstruction strategy to stabilize decoder optimization. The proposed method successfully quantizes both SAM-B and SAM-L to 4 bits, achieving mAP improvements of 14.6% and 6.6% over current state-of-the-art approaches, respectively, thereby significantly enhancing low-bit deployment performance.
0 citationsRead paper
Recent publications
Latest Papers
All for 1-Bit: Towards Genuine 1-Bit Post-Training Quantization for LLMs
为解决大型语言模型部署效率问题，提出All for 1-Bit框架，通过二值化因子分解和层次化夏普利分配方法实现真正的1比特后训练量化。
0 citationsRead paper
TWLA: Achieving Ternary Weights and Low-Bit Activations for LLMs via Post-Training Quantization
This work addresses the challenge of efficiently deploying large language models, which are hindered by high memory and computational demands, while existing ternarization methods struggle to achieve end-to-end low-bit inference due to heavy-tailed activation distributions. To overcome this, the authors propose TWLA, a post-training quantization framework that enables significantly accelerated inference with high accuracy using only 1.58-bit weights and 4-bit activations (W1.58A4). The approach introduces three key innovations: an Euclidean-to-manifold asymmetric ternary quantizer (E2M-ATQ), Kronecker orthogonal trimodal shaping (KOTMS), and inter-layer aware mixed-precision activation allocation (ILA-AMP). Together, these components enable joint optimization of weight ternarization and activation quantization, achieving high-fidelity low-bit representations in manifold space.
0 citationsRead paper
MGVQ: Synergizing Multi-dimensional Sensitivity-Aware and Gradient-Hessian Fusion for Vector Quantization
Existing vector quantization methods for vision-language models struggle to account for cross-modal weight distribution discrepancies and introduce compensation bias due to the neglect of first-order gradient information. To address these limitations, this work proposes MGVQ, a novel framework that jointly models channel sensitivity and gradient–Hessian information for highly efficient ultra-low-bit quantization. The core innovations include a mixed-precision allocation strategy guided by multidimensional sensitivity analysis and a second-order error compensation mechanism integrating gradient embedding with Kronecker and Block-LDL decompositions. Experimental results demonstrate that MGVQ substantially improves performance across multiple state-of-the-art vision-language models—LLaVA-OneVision, InternVL2, and Qwen2-VL—with gains of up to 4.9 accuracy points at 2-bit quantization, achieving 71.4% on InternVL2-26B.
0 citationsRead paper
TORQ: Two-Level Orthogonal Rotation for MXFP4 Quantization
This work addresses the significant accuracy degradation in MXFP4 activation quantization, which stems from a structural mismatch between activation distributions and the block floating-point format. The study reveals, for the first time, that this mismatch manifests as two interrelated issues: inter-block energy imbalance and low intra-block codebook utilization. To mitigate these problems, the authors propose a training-free post-training quantization framework that employs a two-level orthogonal rotation strategy. At the macro level, inter-block rotations based on the Schur-Horn theorem balance activation variances across blocks; at the micro level, intra-block rotations guided by maximum entropy principles enhance codebook efficiency. Evaluated on Qwen3-32B, the method reduces WikiText perplexity to 8.43 and improves average accuracy from 38.40% to 73.63%, substantially narrowing the performance gap between 4-bit floating-point quantized models and their full-precision counterparts.
0 citationsRead paper
CAR-SAM: Cross-Attention Reconstruction for Post-Training Quantization of the Segment Anything Model
Existing post-training quantization methods for the Segment Anything Model (SAM) overlook the cross-attention structure in its decoder, leading to attention information dispersion and unstable reconstruction optimization, which severely degrades segmentation performance at low bit-widths. To address this, this work proposes CAR-SAM, the first quantization framework specifically tailored to SAM’s cross-attention architecture. It introduces a MatMul-aware compensation mechanism to mitigate attention dispersion and a joint cross-attention reconstruction strategy to stabilize decoder optimization. The proposed method successfully quantizes both SAM-B and SAM-L to 4 bits, achieving mAP improvements of 14.6% and 6.6% over current state-of-the-art approaches, respectively, thereby significantly enhancing low-bit deployment performance.
0 citationsRead paper