How good is PAC-Bayes at explaining generalisation?

2025-03-11

AI Summary

This paper investigates whether PAC-Bayes theory can provide meaningful theoretical guarantees for the generalization of deep learning models. Method: The authors systematically analyze necessary conditions for PAC-Bayes bounds to yield non-vacuous generalization guarantees, modeling the risk distribution induced by the prior and deriving prior–posterior bounds grounded in this distribution. Contribution/Results: They prove that the effectiveness of any PAC-Bayes bound fundamentally hinges on the prior’s concentration over low-risk hypotheses—only when the prior assigns sufficient probability mass to such hypotheses does the bound become informative. Crucially, the work establishes for the first time that data-dependent priors cannot circumvent this “prior quality” requirement, thereby challenging the prevailing notion that they automatically explain generalization. The paper demonstrates that achievability of generalization objectives is entirely determined by the prior-induced risk distribution and exposes inherent theoretical limitations of standard data-dependent priors in deep learning. These findings provide critical cautionary insights and delineate fundamental theoretical boundaries for PAC-Bayes methodology.

Technology Category

• Machine Learning: Learning Theory
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

We discuss necessary conditions for a PAC-Bayes bound to provide a meaningful generalisation guarantee. Our analysis reveals that the optimal generalisation guarantee depends solely on the distribution of the risk induced by the prior distribution. In particular, achieving a target generalisation level is only achievable if the prior places sufficient mass on high-performing predictors. We relate these requirements to the prevalent practice of using data-dependent priors in deep learning PAC-Bayes applications, and discuss the implications for the claim that PAC-Bayes ``explains'' generalisation.

Problem

• Analyzing PAC-Bayes bounds for meaningful generalization guarantees.
• Optimal generalization depends on prior-induced risk distribution.
• Evaluating data-dependent priors in deep learning PAC-Bayes applications.

Innovation

• PAC-Bayes bound conditions for generalization
• Optimal generalization depends on prior risk distribution
• Data-dependent priors in deep learning applications