Generalization Certificates for Adversarially Robust Bayesian Linear Regression

2025-02-20

AI Summary

This work addresses the adversarial robustness of Bayesian linear regression. To overcome the lack of theoretical guarantees in existing approaches, we propose a Bregman divergence-based adversarial loss and a novel adversarially robust posterior distribution, establishing— for the first time—the PAC-Bayesian adversarial generalization bound for Bayesian linear regression. This bound is analytically tractable and admits a closed-form solution for adversarial perturbations. Our method integrates exponential-family modeling, generalized Bayesian inference, and PAC-Bayesian theory, avoiding black-box optimization. Experiments on synthetic and real-world datasets demonstrate substantial improvements in adversarial robustness and empirically validate the tightness and effectiveness of the derived generalization bound. Key contributions include: (i) the first formal definition of an adversarially robust posterior; (ii) the first PAC-Bayesian adversarial generalization certificate for Bayesian linear regression; and (iii) an analytically solvable adversarial training framework.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Adversarial robustness of machine learning models is critical to ensuring reliable performance under data perturbations. Recent progress has been on point estimators, and this paper considers distributional predictors. First, using the link between exponential families and Bregman divergences, we formulate an adversarial Bregman divergence loss as an adversarial negative log-likelihood. Using the geometric properties of Bregman divergences, we compute the adversarial perturbation for such models in closed-form. Second, under such losses, we introduce emph{adversarially robust posteriors}, by exploiting the optimization-centric view of generalized Bayesian inference. Third, we derive the emph{first} rigorous generalization certificates in the context of an adversarial extension of Bayesian linear regression by leveraging the PAC-Bayesian framework. Finally, experiments on real and synthetic datasets demonstrate the superior robustness of the derived adversarially robust posterior over Bayes posterior, and also validate our theoretical guarantees.

Problem

• Develops adversarial robustness for Bayesian linear regression.
• Introduces adversarially robust posteriors using optimization-centric view.
• Provides first rigorous generalization certificates via PAC-Bayesian framework.

Innovation

• Adversarial Bregman divergence loss formulation
• Adversarially robust posteriors introduction
• Generalization certificates using PAC-Bayesian framework