Beyond Quadratic Loss: The Stability Phase Diagram of Adam

2026-09-16

AI Summary

研究探讨了Adam优化器中因动量时间尺度不匹配导致的损失尖峰问题，通过分析不同设置下的训练动态，揭示了损失几何与动量参数之间的关系。

Abstract

Loss spikes are recurrent instabilities in neural-network training and can arise from multiple mechanisms. For Adam in particular, macroscopic loss spikes have been linked to optimizer dynamics, yet how its two momentum timescales govern them remains unclear. We investigate this dependence by mapping training dynamics across the $(β_1,β_2)$ plane. Across a range of model--task settings, an approximately linear boundary, $1-β_2=C(1-β_1)$, separates spiky from non-spiky dynamics, whereas a one-dimensional quadratic loss produces approximately cubic slope. A one-dimensional superquadratic loss $L(x)\propto|x|^n$ recovers the near-linear scaling and links the boundary coefficient to the effective loss exponent $n$. We further show that confident cross-entropy losses develop a core--wall landscape comprising a narrow quadratic core followed by a steep wall, which produces effective superquadratic behavior at the scale of an optimizer update. Together, these results connect Adam loss spikes to both the mismatch between momentum timescales and finite-scale superquadratic loss geometry beyond the Hessian.

Problem

• Adam
• loss spikes
• momentum timescales
• training dynamics
• superquadratic loss

Innovation

• Adam optimizer
• loss spikes
• momentum timescales
• superquadratic loss
• core-wall landscape