🤖 AI Summary
This work addresses two finite-sample biases inherent in preconditioned optimizers during language model training: coupling bias between gradient and preconditioner estimates due to minibatch approximation, and systematic bias introduced by nonlinear inversion operations. For the first time, these biases are jointly identified and corrected through a unified single-batch bias-correction framework. The proposed method employs cross-fitting to decouple gradient and preconditioner estimation and incorporates a Delta-method-based variance correction to mitigate nonlinearity-induced bias. Compatible with mainstream optimizers such as AdamW, Sophia, and Shampoo, the approach reduces validation loss by 0.15, 0.07, and 0.11 nats, respectively, in Qwen2.5-0.5B pretraining, while maintaining or improving downstream instruction-tuning performance—demonstrating markedly enhanced training stability and efficiency.
📝 Abstract
Preconditioned optimizers are central to language model training, but their stochastic update rules are usually treated as direct approximations to population preconditioned descent. We show that this view misses two finite-sample biases. First, the gradient and preconditioner are typically estimated from the same minibatch, introducing gradient--preconditioner coupling bias. Second, even when the preconditioner estimate is unbiased, its inverse or inverse-root is generally biased because inversion is nonlinear. We propose a single-batch bias-correction framework that addresses both effects: cross-fitted preconditioning estimates the numerator and preconditioner from independent microbatch groups, while variance-corrected inversion uses microbatch variability to subtract the leading delta-method bias term. The framework applies to diagonal moment, diagonal curvature, and matrix preconditioning methods, instantiated in AdamW, Sophia, and Shampoo. Bias correction reduces held-out pretraining loss on Qwen2.5-0.5B by $0.15$, $0.07$, and $0.11$ nats, respectively; the effects on mixed-quality pretraining and downstream instruction tuning are consistently neutral-to-positive. Together, these results establish bias correction as a practical mechanism for reducing finite-sample update bias and improving the performance of preconditioned optimizers.