Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipe

2026-06-18

AI Summary

This work addresses a critical limitation in existing FP4 pretraining methods, such as E2M1, whose non-uniform numerical representation induces geometric asymmetry, leading to systematic contraction bias and training instability. The study is the first to uncover the geometric origin of this bias and its compounding effect across multiple layers. To mitigate these issues, the authors propose UFP4, a novel FP4 training paradigm based on a uniform 4-bit grid (E1M2/INT4), integrated with randomized Hadamard transforms and directed stochastic rounding. Experiments across model scales from 1.5B to 124B parameters demonstrate that UFP4 consistently outperforms E2M1 baselines with substantially lower loss degradation, thereby establishing the superiority and practical feasibility of uniform formats for FP4 training.

Abstract

FP4 training promises substantial reductions in memory and computation cost for LLM pretraining, yet current FP4 hardware paths and recipes, including NVIDIA Blackwell/Rubin-class systems and AMD MI350-series GPUs, remain centered on E2M1 data elements. In this study, we identify a fundamental limitation of that choice: non-uniform formats such as E2M1 inherently suffer from Shrinkage Bias, a systematic negative rounding error caused by the geometric asymmetry of their representable bins. We show that this bias accumulates multiplicatively across layers and is amplified by the Random Hadamard Transform (RHT), providing a unified explanation for the training instability observed in existing E2M1-based FP4 recipes. In contrast, uniform grids (E1M2/INT4) bypass this grid-geometry error and better convert the improved bucket utilization from RHT into higher quantization quality. Based on this finding, we propose UFP4, a uniform 4-bit training recipe that applies RHT to all three training GEMMs while restricting stochastic rounding to dY alone. On Dense 1.5B, MoE 7.9B, and MoE 124B long-run pretraining, UFP4 consistently achieves lower BF16-relative loss degradation than strong E2M1-based baselines, supported by scaling-law analysis and ablation studies. Our results suggest that future accelerators should support E1M2/INT4-style uniform 4-bit grids as first-class training primitives alongside E2M1.

Problem

• Shrinkage Bias
• FP4 training
• E2M1 format
• geometric asymmetry
• training instability

Innovation

• Shrinkage Bias
• Uniform FP4
• Random Hadamard Transform
• E1M2
• UFP4