VASSO: Variance Suppression for Sharpness-Aware Minimization

2025-09-02

AI Summary

While SAM improves model generalization, its adversarial perturbations are overly permissive—exhibiting insufficient sharpness awareness and consequently limited generalization. This work proposes Variance-Aware SAM (VA-SAM), the first method to explicitly constrain the variance of adversarial perturbations within the SAM framework, thereby enhancing robustness and consistency. We theoretically establish that this variance-suppression mechanism stabilizes optimization trajectories and tightens generalization bounds. VA-SAM introduces only a lightweight variance regularization term, incurring no additional computational overhead beyond standard SAM. Evaluated across over ten benchmarks—including image classification, semantic segmentation, and language modeling—VA-SAM consistently outperforms SAM and leading variants (e.g., ASAM, GSAM), achieving average Top-1 accuracy gains of 0.8–1.5 percentage points. These results demonstrate VA-SAM’s superior trade-off between generalization performance and computational efficiency.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Security and Privacy: Large-scale security measurements

Abstract

Sharpness-aware minimization (SAM) has well-documented merits in enhancing generalization of deep neural network models. Accounting for sharpness in the loss function geometry, where neighborhoods of `flat minima' heighten generalization ability, SAM seeks `flat valleys' by minimizing the maximum loss provoked by an adversarial perturbation within the neighborhood. Although critical to account for sharpness of the loss function, in practice SAM suffers from `over-friendly adversaries,' which can curtail the outmost level of generalization. To avoid such `friendliness,' the present contribution fosters stabilization of adversaries through variance suppression (VASSO). VASSO offers a general approach to provably stabilize adversaries. In particular, when integrating VASSO with SAM, improved generalizability is numerically validated on extensive vision and language tasks. Once applied on top of a computationally efficient SAM variant, VASSO offers a desirable generalization-computation tradeoff.

Problem

• Stabilizes adversarial perturbations in sharpness-aware minimization
• Improves generalization of deep neural network models
• Suppresses variance to prevent over-friendly adversaries

Innovation

• VASSO stabilizes adversaries through variance suppression
• Integrates variance suppression with SAM algorithm
• Improves generalization-computation tradeoff in neural networks