Width-Robust Learnability in Mean-Field Bayesian Neural Networks

2026-07-06

AI Summary

This work investigates whether infinite-width Bayesian neural networks, under the mean-field (critical) scaling regime, retain the computational complexity inductive bias inherent to finite-width networks. By introducing the notion of “reduced entropy” to characterize the learnability of target functions and employing a subsampling strategy that distinguishes active from inert components, the authors eliminate spurious width dependencies and establish a width-robust theory of learnability. The main contribution is a rigorous proof—established for the first time—that on the Boolean cube, a family of target functions is polynomially sample-learnable by an infinite-width network if and only if its reduced entropy is polynomially bounded, which is equivalent to the existence of a finite-width network of polynomial size achieving exact generalization.

Abstract

Infinite-width limits are a standard way to reason about neural networks, but it is not automatic that the limiting learner has the same complexity-theoretic inductive bias as large finite networks. We study this question for Bayesian neural networks at the mean-field, or critical feature-learning, scaling. The central quantity is the \emph{reduced entropy} \[ s_\infty(y,\varepsilon)=\limsup_N -\frac{1}{N}\log π_N^0(L\le \varepsilon), \] the intensive prior cost of representing a target function $y$ to population mean-squared error $\varepsilon$. Our main result is a width-robust learnability theorem. At fixed depth, a family of Boolean-cube targets is learnable from polynomially many samples at infinite width if and only if it is learnable at polynomial width, if and only if its reduced entropy is polynomially bounded. Equivalently, up to polynomial slack in accuracy, the Bayesian mean-field learner generalizes exactly on the targets that can be represented by polynomial-size networks. The forward direction is proved by a form of subsampling: from the infinitely many hidden neurons in the mean-field solution, one can select polynomially many representatives and still preserve the learned function on every input simultaneously. At the critical scaling this subsampling has both an ``active'' component, which keeps the data-dependent low-dimensional statistics, and a ``lazy'' component, which resamples the entropy-dominated directions from the prior. Thus the infinite-width mean-field limit gives a clean analytic description of learning without introducing spurious width-dependent generalization power.

Problem

• width-robust learnability
• Bayesian neural networks
• mean-field limit
• reduced entropy
• inductive bias

Innovation

• width-robust learnability
• mean-field Bayesian neural networks
• reduced entropy
• infinite-width limit
• subsampling