Leveraging Extragradient for Effective Sharpness-Aware Minimization in Deep Learning

2026-07-07

AI Summary

This work proposes EISAM, a novel optimizer that integrates extragradient methods with Sharpness-Aware Minimization (SAM) to address the tendency of conventional optimizers to converge to sharp minima, which harms model generalization. EISAM employs a prediction step to explore the geometry of the loss landscape and a perturbation step that jointly updates parameters with the base optimizer, enabling more efficient convergence to flat minima. This design substantially reduces sensitivity to the perturbation radius, enhancing robustness and simplifying hyperparameter tuning. Extensive experiments demonstrate that EISAM consistently outperforms SGD, Adam, and SAM across diverse models and benchmark datasets, achieving higher test accuracy and improved training efficiency. The study also provides practical guidelines for hyperparameter configuration.

Abstract

Generalization remains a pivotal challenge in deep learning, where traditional optimizers like Stochastic Gradient Descent (SGD) often converge to sharp minima, leading to overfitting and reduced performance on unseen data. Building on Sharpness-Aware Minimization (SAM), for seeking flat minima associated with improved generalization, we propose the Extragradient-Inspired Sharpness-Aware Minimization (EISAM), a novel optimizer that enhances generalization via the extragradient technique. EISAM uses a two-step update process: a prediction step investigating the geometry of the loss landscape and a perturbation step that refines updates with a base optimizer. This approach achieves better generalization performance than SAM. Crucially, EISAM reduces sensitivity to the perturbation radius, enhancing robustness, and simplifying the tuning across diverse settings. Extensive experiments on benchmark datasets demonstrate that EISAM consistently outperforms SGD, Adaptive Moment Estimation (Adam), and SAM in test accuracy and training efficiency across various architectures. Theoretical analysis further confirms that EISAM tightens the generalization bound by steering parameters toward flatter minima with reduced curvature. Accompanied by a thorough hyperparameter analysis, EISAM offers practical tuning guidance, establishing it as a robust, scalable, and broadly applicable optimization solution that advances both the theory and practice in deep learning.

Problem

• generalization
• sharp minima
• overfitting
• flat minima
• deep learning

Innovation

• Extragradient
• Sharpness-Aware Minimization
• Flat Minima
• Generalization
• Optimization