Anon: Extrapolating Optimizer Adaptivity Across the Real Spectrum

2026-05-04

AI Summary

This work addresses the long-standing generalization gap of adaptive optimizers—such as Adam—on classical architectures like CNNs, where they underperform compared to SGD despite excelling in large-scale models. The core issue lies in their fixed adaptivity, which struggles to accommodate diverse optimization landscapes. To resolve this, we propose Anon, the first optimizer enabling continuously tunable and extrapolatable adaptivity by interpolating between SGD and Adam behaviors over the real number domain. Anon further introduces an Incremental Delayed Update (IDU) mechanism that replaces conventional maximum tracking, ensuring convergence across both convex and non-convex settings. Extensive experiments demonstrate that Anon consistently outperforms state-of-the-art optimizers on image classification, diffusion models, and language modeling tasks, validating adaptivity as a principled, adjustable design dimension in optimization.

Abstract

Adaptive optimizers such as Adam have achieved great success in training large-scale models like large language models and diffusion models. However, they often generalize worse than non-adaptive methods, such as SGD on classical architectures like CNNs. We identify a key cause of this performance gap: adaptivity in pre-conditioners, which limits the optimizer's ability to adapt to diverse optimization landscapes. To address this, we propose Anon (Adaptivity Non-restricted Optimizer with Novel convergence technique), a novel optimizer with continuously tunable adaptivity in R, allowing it to interpolate between SGD-like and Adam-like behaviors and even extrapolate beyond both. To ensure convergence across the entire adaptivity spectrum, we introduce incremental delay update (IDU), a novel mechanism that is more flexible than AMSGrad's hard max-tracking strategy and enhances robustness to gradient noise. We theoretically establish convergence guarantees under both convex and non-convex settings. Empirically, Anon consistently outperforms state-of-the-art optimizers on representative image classification, diffusion, and language modeling tasks. These results demonstrate that adaptivity can serve as a valuable tunable design principle, and Anon provides the first unified and reliable framework capable of bridging the gap between classical and modern optimizers and surpassing their advantageous properties.

Problem

• adaptive optimizers
• generalization gap
• optimization landscapes
• pre-conditioners
• SGD vs Adam

Innovation

• adaptive optimizer
• continuously tunable adaptivity
• incremental delay update
• convergence guarantee
• unified optimization framework