Pretraining large language models with MXFP4

2026-05-10

AI Summary

This work investigates the instability of full-pipeline FP4 training in large language models, which often diverges due to weight gradient quantization despite stable forward activations and activation gradients. Through a systematic analysis of MXFP4 quantization across all training stages on Llama 3.1-8B, the study identifies structured microscaling errors—not insufficient randomness—as the primary cause of optimization divergence in weight gradient quantization. To mitigate this issue, the authors introduce a deterministic Hadamard rotation that effectively suppresses such errors, enabling stable native FP4 end-to-end training on AMD Instinct MI355X GPUs. Experiments demonstrate that quantizing only forward passes and activation gradients has minimal impact on convergence, whereas incorporating the Hadamard rotation allows successful completion of full pretraining on the C4 dataset.

Abstract

Why does full-pipeline FP4 training of large language models often diverge, even when forward activations and activation gradients remain stable? We address this question through a controlled study of MXFP4 quantization in transformer training, progressively enabling FP4 across forward propagation (Fprop), activation gradients (Dgrad), and weight gradients (Wgrad) while holding all other factors fixed. In full pretraining of Llama 3.1-8B on the C4 dataset, we observe that quantizing Wgrad is the primary driver of convergence degradation, whereas FP4 in Fprop and Dgrad alone introduces only modest additional token requirements. To interpret this behavior, we evaluate both structured and stochastic interventions under a controlled experimental setting. We find that stochastic rounding and randomized Hadamard rotations fail to stabilize training once Wgrad is quantized, whereas deterministic Hadamard rotations consistently restore stable optimization. These results suggest that FP4 training instability is driven by structured micro-scaling errors along sensitive gradient paths, rather than by insufficient stochasticity. We run experiments with native MXFP4 support on AMD Instinct MI355X GPUs, enabling controlled investigation of these effects without reliance on software emulation.

Problem

• FP4 training
• convergence instability
• weight gradients
• quantization
• large language models

Innovation

• MXFP4
• weight gradient quantization
• Hadamard rotation
• low-bit training
• training stability