Smoothness-Based Derandomization of PAC-Bayes Bounds

2026-06-17

AI Summary

This work addresses the problem of establishing high-probability generalization bounds for deterministic predictors, particularly in settings with smooth loss functions. By leveraging Jacobian and Hessian smoothness of both the loss function and the predictor class, the authors extend PAC-Bayes bounds—traditionally applicable to Gibbs (randomized) predictors—to deterministic predictors evaluated at the posterior mean. This extension effectively controls the Jensen gap and integrates Rademacher complexity analysis. The proposed framework yields a unified generalization bound based on parameter flatness, accommodating both bounded and unbounded losses, and employs BatchNorm weight folding for computationally efficient implementation. Experiments on CIFAR-10 validate the efficacy of the derived regularization term and elucidate the interplay among batch size, solution flatness, and generalization performance.

Abstract

We study PAC-Bayes derandomization for smooth loss functions. Our goal is to obtain generalization bounds that hold with high probability for deterministic predictors by exploiting smoothness properties of both the loss and the predictor class. We show that passing from the Gibbs predictor to the deterministic predictor at the posterior mean has a precise cost, given by the generalization gap of the Jensen gap class. We control this class through its Rademacher complexity, leading to bounds for deterministic predictors that involve flatness quantities expressed in terms of parameter Jacobians and Hessians of the score map. The framework applies to both bounded and unbounded smooth loss functions, and we specialize the results to linear predictors and smooth neural networks. Finally, the Jacobian and Hessian quantities appearing in the theory motivate a practical regularizer. For BatchNorm networks, we compute this regularizer with respect to effective BatchNorm weights obtained by folding the BatchNorm transformation into the adjacent affine weights. Experiments on CIFAR-10 illustrate the behavior of this regularizer under different batch sizes.

Problem

• PAC-Bayes
• derandomization
• smooth loss
• generalization bounds
• deterministic predictors

Innovation

• PAC-Bayes derandomization
• smooth loss functions
• Jensen gap
• Rademacher complexity
• flatness regularizer