Revisiting the Volume Hypothesis

2026-06-30

AI Summary

This study investigates the mechanisms underlying the strong generalization of over-parameterized deep neural networks, with a focus on the “volume hypothesis”—the conjecture that regions of low loss exhibiting good generalization occupy larger volumes in weight space. The authors present the first systematic analysis of how training data size influences this hypothesis, employing the replica-exchange Wang–Landau algorithm to estimate the joint density of states of training and test accuracy in binary neural networks. This enables quantitative characterization of the volumes associated with different generalization behaviors in high-dimensional weight space. Their experiments reveal that as dataset size increases, the generalization advantage of gradient descent over random sampling diminishes. The findings validate the volume hypothesis in large-data regimes across multiple architectures and datasets, offering a unified explanation for previously seemingly contradictory empirical observations.

Abstract

Modern deep neural networks often contain far more parameters than needed to fit their training data, yet they achieve impressive generalization. A common explanation for this success is the implicit bias of stochastic gradient descent (SGD). An alternative volume hypothesis posits that, within low training-loss regions, loss-landscape basins leading to strong generalization occupy much larger regions of weight space than basins that generalize poorly, and therefore SGD is simply more likely to land in the former. Recent experimental explorations of this idea present seemingly contradictory results. While in one set of experiments randomly sampling the network weights until achieving zero training error yielded poor generalization, molecular dynamics density estimates supported the volume hypothesis. We observe that these experiments were performed at different dataset size regimes, and explore an intermediate regime using the Replica Exchange Wang-Landau algorithm to estimate the joint density of states over training and test accuracies in binary networks. Across several architectures and datasets, we show that the generalization advantage of gradient learning over random sampling training generally diminishes as the training data size grows, suggesting a resolution of the paradox.

Problem

• volume hypothesis
• generalization
• stochastic gradient descent
• random sampling
• dataset size

Innovation

• volume hypothesis
• Replica Exchange Wang-Landau
• generalization
• density of states
• binary neural networks