On the Epistemic Uncertainty of Overparametrized Neural Networks

2026-05-24

AI Summary

Conventional wisdom holds that epistemic uncertainty vanishes as data increases; however, in over-parameterized neural networks, significant residual uncertainty persists at the parameter level even when the underlying function is fully identified. This work systematically investigates the structure of such uncertainty arising from parameter non-identifiability, distinguishing between discrete sources (e.g., neuron permutation) and continuous symmetries. Focusing on the Bayesian posterior geometry of single-hidden-layer ReLU networks, the study combines theoretical analysis with empirical validation to uncover a novel mechanism by which epistemic uncertainty fails to diminish with more data. It rigorously disentangles functional identifiability from parameter identifiability and precisely characterizes the origin and form of the residual uncertainty, thereby advancing the understanding of uncertainty quantification in Bayesian neural networks.

Abstract

Epistemic uncertainty is often viewed as a reducible uncertainty that vanishes with increasing data. This perspective implicitly assumes parameter identifiability and equates epistemic uncertainty with predictive variability. In overparametrized neural networks, however, model parameters are typically non-identifiable due to symmetries and redundant representations. As a consequence, substantial parameter uncertainty can persist even when the underlying function is fully identified. In this work, we analyze epistemic uncertainty through the lens of non-identifiability and characterize both discrete and continuous sources of residual uncertainty. Focusing on one-hidden-layer ReLU networks, we thoroughly analyze the resulting posterior structure and validate our theoretical insights through empirical studies.

Problem

• epistemic uncertainty
• overparametrized neural networks
• parameter identifiability
• non-identifiability
• residual uncertainty

Innovation

• epistemic uncertainty
• overparametrized neural networks
• parameter non-identifiability
• posterior structure
• ReLU networks