Preconditioned Sharpness-Aware Minimization: Unifying Analysis and a Novel Learning Algorithm

2025-01-11

AI Summary

Sharpness-Aware Minimization (SAM) often induces adversarial model degradation during training—particularly degrading generalization when loss landscapes are flat. To address this, we propose preSAM, a preconditioned unified analytical framework that, for the first time, models mainstream SAM variants as instances of a generic preconditioned optimization scheme. Building upon preSAM, we design infoSAM, a noise-aware algorithm that performs adaptive gradient correction via dynamic noise estimation along the gradient direction. We establish its theoretical convergence under standard assumptions. Extensive experiments across diverse benchmarks—including image classification and semantic segmentation—demonstrate that infoSAM consistently outperforms SAM and its variants by significant margins. Our results empirically validate that jointly optimizing for loss landscape flatness and noise robustness synergistically enhances generalization performance.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Learning & Optimization for CV

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Targeting solutions over `flat' regions of the loss landscape, sharpness-aware minimization (SAM) has emerged as a powerful tool to improve generalizability of deep neural network based learning. While several SAM variants have been developed to this end, a unifying approach that also guides principled algorithm design has been elusive. This contribution leverages preconditioning (pre) to unify SAM variants and provide not only unifying convergence analysis, but also valuable insights. Building upon preSAM, a novel algorithm termed infoSAM is introduced to address the so-called adversarial model degradation issue in SAM by adjusting gradients depending on noise estimates. Extensive numerical tests demonstrate the superiority of infoSAM across various benchmarks.

Problem

• Sharpness-Aware Minimization
• Adversarial Robustness
• Deep Learning Optimization

Innovation

• infoSAM
• Sharpness-Aware Minimization
• Adversarial Robustness