Dataset Complexity Shapes Finite-Distance Loss Geometry in Neural Networks

2026-08-23

AI Summary

研究通过局部熵和自适应序列蒙特卡洛方法，探讨数据集复杂性如何影响神经网络损失景观的几何结构。

Abstract

Finite datasets can share the same size and low-order statistics while differing strongly in structural complexity. We connect this dataset complexity to loss-landscape geometry by pairing local label mixing across neighborhood scales with local entropy around trained neural-network solutions. Adapted from the Franz--Parisi construction in spin-glass theory, local entropy measures the effective volume of low-loss, solution-like parameter configurations at each distance from a reference. We estimate it in finite networks using adaptive sequential Monte Carlo. In a controlled synthetic sweep, greater dataset complexity produces a larger decrease in local entropy near the reference. Farther away, its radial derivative becomes weak and nearly common across conditions. Dataset complexity therefore changes where the effective solution volume contracts, rather than making it decrease uniformly faster. Experiments on real image data show the same qualitative trend, with label randomization further amplifying the effect. These results show that dataset structure shapes how low-loss neighborhoods are organized across finite distances from trained solutions.

Problem

• dataset complexity
• loss-landscape geometry
• local entropy
• neural networks
• finite distances

Innovation

• local entropy
• dataset complexity
• loss landscape geometry
• finite neural networks
• adaptive sequential Monte Carlo