PAC-Bayes Bounds for Gibbs Posteriors via Singular Learning Theory

2026-04-18

AI Summary

This work addresses the looseness of classical generalization bounds based on metric entropy for over-parameterized and singular models. It introduces, for the first time, a synthesis of singular learning theory with the PAC-Bayes framework, deriving a data-dependent, non-asymptotic, and tight generalization error bound by analyzing the marginal likelihood of the Gibbs posterior. The resulting bound adapts to both the underlying data structure and the intrinsic complexity of the model, and is analytically tractable in practical settings such as low-rank matrix completion and ReLU neural networks. This approach yields significantly sharper guarantees than traditional complexity-based bounds, offering more realistic finite-sample theoretical assurances for modern machine learning models.

Technology Category

• Machine Learning: Learning Theory
• Natural Language Processing: Learning & Optimization for NLP
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

We derive explicit non-asymptotic PAC-Bayes generalization bounds for Gibbs posteriors, that is, data-dependent distributions over model parameters obtained by exponentially tilting a prior with the empirical risk. Unlike classical worst-case complexity bounds based on uniform laws of large numbers, which require explicit control of the model space in terms of metric entropy (integrals), our analysis yields posterior-averaged risk bounds that can be applied to overparameterized models and adapt to the data structure and the intrinsic model complexity. The bound involves a marginal-type integral over the parameter space, which we analyze using tools from singular learning theory to obtain explicit and practically meaningful characterizations of the posterior risk. Applications to low-rank matrix completion and ReLU neural network regression and classification show that the resulting bounds are analytically tractable and substantially tighter than classical complexity-based bounds. Our results highlight the potential of PAC-Bayes analysis for precise finite-sample generalization guarantees in modern overparameterized and singular models.

Problem

• PAC-Bayes
• Gibbs posteriors
• generalization bounds
• overparameterized models
• singular learning theory

Innovation

• PAC-Bayes
• Gibbs posterior
• singular learning theory
• overparameterized models
• generalization bounds