S2O: Enhancing Adversarial Training with Second-Order Statistics of Weights

2026-03-01

AI Summary

This work addresses the challenge of simultaneously achieving robustness and generalization in deep neural networks under adversarial attacks by proposing a second-order statistics optimization method (S²O) grounded in PAC-Bayesian theory. S²O explicitly models the stochasticity of network weights during adversarial training and optimizes their covariance structure, thereby overcoming the restrictive assumption of weight independence inherent in conventional PAC-Bayesian frameworks. The approach yields a significantly tighter robust generalization bound and enhances model performance without requiring additional components. Furthermore, S²O effectively boosts both robustness and generalization when integrated with state-of-the-art adversarial training techniques.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Search and Optimization: Adversarial Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Adversarial training has emerged as a highly effective way to improve the robustness of deep neural networks (DNNs). It is typically conceptualized as a min-max optimization problem over model weights and adversarial perturbations, where the weights are optimized using gradient descent methods, such as SGD. In this paper, we propose a novel approach by treating model weights as random variables, which paves the way for enhancing adversarial training through \textbf{S}econd-Order \textbf{S}tatistics \textbf{O}ptimization (S$^2$O) over model weights. We challenge and relax a prevalent, yet often unrealistic, assumption in prior PAC-Bayesian frameworks: the statistical independence of weights. From this relaxation, we derive an improved PAC-Bayesian robust generalization bound. Our theoretical developments suggest that optimizing the second-order statistics of weights can substantially tighten this bound. We complement this theoretical insight by conducting an extensive set of experiments that demonstrate that S$^2$O not only enhances the robustness and generalization of neural networks when used in isolation, but also seamlessly augments other state-of-the-art adversarial training techniques. The code is available at https://github.com/Alexkael/S2O.

Problem

• adversarial training
• robustness
• PAC-Bayesian
• weight dependence
• generalization bound

Innovation

• Second-Order Statistics
• Adversarial Training
• PAC-Bayesian
• Robust Generalization
• Weight Covariance