CurveFP: Rational-Radix Logarithmic Datatypes with Closed Products for Language Models

2026-08-07

AI Summary

Existing low-precision floating-point formats struggle to simultaneously balance dynamic range, local precision, and computational efficiency due to their rigid multiplicative structure, despite preserving scalar fidelity. This work proposes CurveFP, a rationally based logarithmic data type with multiplicative closure that simplifies multiplication to sign XOR and integer index updates by interleaving logarithmic curves with compact block scaling. It further introduces an algebraically closed non-zero product mechanism to jointly modulate dynamic range and resolution. Using 8-bit E4C3/E5C2 formats for training and a 7-bit E3C3 format for inference—augmented with finite-phase counting to schedule accumulation—CurveFP’s 7-bit variant surpasses FP8 in perplexity with one fewer bit, while its 8-bit configuration consistently achieves lower normalized mean squared error (NMSE) across all GEMM benchmarks and closely matches BF16 performance in both pretraining and downstream tasks.

Abstract

Low-precision datatypes reduce language-model cost, but most formats optimize scalar fidelity while leaving the arithmetic induced by their products unchanged. We introduce CurveFP, a closed-product codebook family that distributes quantized magnitudes across interleaved logarithmic curves under compact block scales. A rational radix tunes dynamic range against local resolution, while uniform curve indices make every nonzero product algebraically closed. Product formation becomes an exact sign XOR and integer-index update, and a derived finite phase count determines the accumulation schedule. We instantiate this algebra as CurveFP eight E4C3/E5C2 for training and CurveFP seven E3C3 for compact deployment. In evaluation, CurveFP seven beats tensor-wise FP8 perplexity on four 7B--9B models with one fewer element bit and stays within 1.32\% of native quality. CurveFP eight lowers operand NMSE in all 36 paired forward and backward GEMM comparisons. Across three matched 128.3M-parameter triplets, every mode completes 3B-token pretraining per seed; CurveFP eight reaches mean BF16-inference perplexity 22.5366 versus 22.5407 for FP8 and incurs a lower format-induced penalty in all three seeds. A 36-cell downstream matrix finds lower WikiText-103 perplexity for the CurveFP eight-trained checkpoints in all 12 seed-format comparisons, with mixed PG-19 and task deltas. Together, these results establish CurveFP as an arithmetic co-design that combines FP8-class numerical behavior, seven-bit inference, and a substantially simpler product path.

Problem

• low-precision datatypes
• language models
• arithmetic closure
• numerical fidelity
• product operations

Innovation

• CurveFP
• closed-product
• rational-radix
• logarithmic quantization
• low-precision arithmetic