Uncertainty Estimation and Generalization Bounds for Modern Deep Learning

2026-06-11

AI Summary

Despite the strong predictive performance of deep neural networks, their generalization mechanisms and uncertainty quantification lack rigorous theoretical foundations. This work proposes a unified probabilistic framework that integrates Bayesian inference, function-space modeling, and large deviation theory, offering the first synthesis of diversity, smoothness, and stochasticity—the three key drivers of generalization—within the PAC-Bayes and large deviation paradigms. The core contributions include the Deep Variational Implicit Process (DVIP) model and two efficient post-hoc calibration methods, VaLLA and FMGP, which directly calibrate uncertainty estimates of pre-trained deterministic networks. These advances not only enable practical uncertainty quantification but also provide a theoretical explanation for the surprisingly good generalization observed in over-parameterized neural networks.

Abstract

This thesis investigates how Bayesian principles can deepen our understanding of modern deep learning systems. While neural networks achieve remarkable predictive performance, their ability to generalize and to quantify uncertainty remains only partly understood. This thesis approaches this challenge from both methodological and theoretical angles: unifying Bayesian inference, function-space modeling, and large-deviation theory under a common probabilistic perspective. On the methodological side, the thesis introduces the Deep Variational Implicit Process (DVIP), a scalable Bayesian framework that extends implicit processes to deep architectures. Complementing this, two post-hoc methods -- the Variational Linearized Laplace Approximation (VaLLA) and the Fixed-Mean Gaussian Process (FMGP) -- are proposed to equip pretrained deterministic networks with calibrated uncertainty estimates. The theoretical contributions focus on one of the central open questions in modern machine learning: why do large, over-parameterized neural networks generalize so well? To address this, the thesis develops a unified probabilistic framework that connects three key mechanisms -- diversity, smoothness, and stochasticity -- within the language of PAC-Bayesian and large-deviation theory.

Problem

• uncertainty estimation
• generalization bounds
• deep learning
• over-parameterized neural networks
• Bayesian inference

Innovation

• Deep Variational Implicit Process
• Uncertainty Estimation
• PAC-Bayesian Theory
• Over-parameterized Neural Networks
• Variational Linearized Laplace Approximation