Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization

2026-06-15

AI Summary

This work addresses the diminishing pretraining acceleration advantage of existing matrix-based optimizers—such as Muon—over AdamW as model and data scales increase. To mitigate this, the authors propose Hyperball, an optimizer wrapper that, for the first time, constrains both weight matrices and their updates to have constant Frobenius norm via a hyperspherical normalization mechanism, thereby stabilizing optimization dynamics. This approach effectively decouples the angular learning rate, enhancing its transferability across diverse model architectures while remaining compatible with base optimizers like Adam and Muon. Empirical results on a 1.2B-parameter Qwen3-style model demonstrate that Muon combined with Hyperball achieves a 20–30% token-equivalent training speedup compared to standard weight decay baselines.

Abstract

Matrix based optimizers such as Muon can substantially speed up language model pretraining, but their gains over AdamW are observed to shrink as model size and data scale grow when using standard constant decoupled weight decay. We propose Hyperball, a simple optimizer wrapper that addresses this issue. Given a base optimizer such as Adam or Muon, Hyperball sets the Frobenius norms of weight matrices and their corresponding optimizer updates to fixed constants. On Qwen3 style models up to 1.2B parameters, Muon Hyperball achieves 20--30% token equivalent speedup over weight decay baselines. Hyperball also improves learning rate transfer across widths and depths compared to decoupled weight decay. This method is motivated by prior theory showing that training with weight decay leads to an equilibrium weight norm that only depends on the training hyperparameters. Through this mechanism, the weight decay then decides the angular learning rate, i.e. how fast the direction of the weight matrix changes.

Problem

• pretraining
• optimizer
• weight decay
• language model
• scaling

Innovation

• Hyperball
• optimizer
• weight decay
• Frobenius norm
• language model pretraining