A PAC-Bayesian Link Between Generalisation and Flat Minima

2024-02-13

arXiv.org

AI Summary

Over-parameterized deep learning models generalize well despite lacking rigorous theoretical explanations—a foundational challenge in modern machine learning. Method: This paper establishes a direct link between optimization dynamics and generalization by attributing generalization performance to the flatness of the attained local minima—regions where the loss function remains approximately constant. Leveraging the PAC-Bayes framework combined with Poincaré and Log-Sobolev inequalities, we derive a tight, **dimension-free** generalization upper bound that explicitly depends on the flatness of the minimum (quantified via the Hessian spectrum or gradient variance), rather than the number of model parameters. Contribution/Results: Our bound circumvents the strong dependence on high-dimensional parameter counts inherent in classical generalization bounds. It provides the first rigorous theoretical foundation for the empirical observation that gradient-based optimization implicitly favors flat minima—and thereby enhances generalization—revealing a novel mechanism by which optimization shapes generalization.

Technology Category

• Machine Learning: Learning Theory
• Natural Language Processing: Learning & Optimization for NLP
• Computer Vision: Learning & Optimization for CV

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Modern machine learning usually involves predictors in the overparameterised setting (number of trained parameters greater than dataset size), and their training yields not only good performance on training data, but also good generalisation capacity. This phenomenon challenges many theoretical results, and remains an open problem. To reach a better understanding, we provide novel generalisation bounds involving gradient terms. To do so, we combine the PAC-Bayes toolbox with Poincar'e and Log-Sobolev inequalities, avoiding an explicit dependency on the dimension of the predictor space. Our results highlight the positive influence of flat minima (being minima with a neighbourhood nearly minimising the learning problem as well) on generalisation performance, involving directly the benefits of the optimisation phase.

Problem

• Explores generalisation in overparameterised machine learning
• Links flat minima to improved generalisation performance
• Uses PAC-Bayes and inequalities to bound generalisation

Innovation

• PAC-Bayes toolbox integration
• Poincaré and Log-Sobolev inequalities
• Flat minima enhance generalization