FAAR: Format-Aware Adaptive Rounding for NVFP4

2026-03-23

AI Summary

This work addresses the significant performance degradation of existing quantization methods when applied to non-uniform low-bit formats such as NVFP4, which stems from their neglect of the format’s intrinsic numerical grid structure, leading to large rounding errors. To overcome this limitation, we propose Format-Aware Adaptive Rounding (FAAR), a novel strategy that explicitly incorporates NVFP4’s non-uniform quantization grid into the rounding optimization process for the first time. This is complemented by a lightweight two-stage Format Alignment (2FA) fine-tuning mechanism, enabling efficient approximation of the theoretically optimal quantization. Evaluated on Llama3-1B and Qwen3-1.7B, our approach achieves WikiText-2 perplexities of 12.60 and 21.27, respectively, and consistently outperforms prior methods across zero-shot tasks, with only approximately 4 additional GPU-hours of training overhead.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: (Large) Language Models
• Computer Vision: Diffusion Models for Vision

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Deploying large language models (LLMs) on edge devices requires extremely low-bit quantization. Ultra-low precision formats such as NVFP4 offer a promising solution for reducing memory footprint and accelerating computation. However, existing quantization methods typically rely on conventional rounding strategies and fail to account for the non-uniformity of the NVFP4 numerical grid, resulting in suboptimal rounding decisions and amplified quantization errors. To address this, we propose Format-Aware Adaptive Rounding (FAAR), a learnable rounding strategy tailored for the NVFP4 format. Unlike conventional quantization paradigms, FAAR explicitly incorporates the non-uniform NVFP4 grid into the optimization process. By adaptively adjusting rounding decisions guided by loss gradients, our method effectively approximates the theoretically optimal quantization. To complement FAAR, we introduce a 2-stages Format Alignment (2FA) fine-tuning scheme that aligns LLM parameters layer-by-layer to the NVFP4 numerical space, further narrowing the performance gap. Remarkably, this learnable optimization incurs a minimal training overhead of only 4 GPU hours on Llama3-1B. Extensive experiments demonstrate the effectiveness of our approach. Compared with Round-to-Nearest (RTN), our method reduces perplexity on WikiText-2 from 14.28 to 12.60 on Llama3-1B and from 23.06 to 21.27 on Qwen3-1.7B. Additionally, our method consistently outperforms state-of-the-art approaches across various zero-shot downstream tasks.

Problem

• quantization
• NVFP4
• rounding error
• non-uniform grid
• large language models

Innovation

• Format-Aware Adaptive Rounding
• NVFP4
• non-uniform quantization
• learnable rounding
• low-bit LLM deployment