M+Adam: Low-Precision Training via Additive-Multiplicative Optimization

2026-07-12

AI Summary

This work addresses the challenge of optimizer stagnation and performance degradation in low-precision training, which is exacerbated by rounding errors—particularly when weight magnitudes are large or signs flip. To mitigate this, the authors propose M+Adam, the first optimizer to integrate both additive and multiplicative update mechanisms. The additive component effectively handles small-magnitude weights and sign changes, while the multiplicative component preserves optimization efficacy in regions of large weight magnitudes; together, they compensate for each other’s limitations near zero and at large values. Under standard smoothness assumptions, the algorithm exhibits monotonic descent. Evaluated within a low-precision training framework using BF16/FP8/FP4 master weights, M+Adam significantly enhances training stability and model performance across architectures ranging from 60M to 1B parameters and under Chinchilla-compliant compute budgets from 1× to 8×, demonstrating strong applicability to LLaMA-style pretraining.

Abstract

Training with quantized weights can reduce costs but often results in degraded accuracy, especially when optimization is carried out in low precision, without storing high-precision copies. We identify a key failure mode: under low precision, standard optimizers can get stuck and not make progress, especially at large weight magnitudes due to coarse mantissa resolution. To overcome this, multiplicative updates have been previously proposed, in place of additive updates in standard optimizers. While successful under extremely low precision, such as under the logarithmic number system, they suffer from failures near zero and across sign changes. The failure modes of additive and multiplicative updates are therefore complementary. To exploit this, we propose M+Adam, which combines both update types: additive steps handle sign changes and small magnitudes, while multiplicative steps ensure progress at large magnitudes when additive updates are zeroed out under rounding. We prove monotone descent for M+Adam under standard smoothness assumptions. Across LLaMA-style pretraining with 60M-1B models, 1x-8x Chinchilla budgets, and using only BF16, FP8, and FP4 master weights, M+Adam consistently improves low-precision training.

Problem

• low-precision training
• quantized weights
• additive updates
• multiplicative updates
• optimizer failure

Innovation

• low-precision training
• multiplicative updates
• additive updates
• M+Adam
• quantized optimization