🤖 AI Summary
This paper addresses the “black-box” nature and poorly understood generalization mechanisms of neural networks by proposing the Local Extrema Dynamic Mapping Hypothesis: generalization arises from a model’s ability to adaptively map training data onto local extrema of the loss landscape. Theoretical analysis reveals that the number of local extrema scales positively with the number of trainable parameters. Leveraging this insight, we design the Extrema Incremental Algorithm (EIA), which replaces conventional backpropagation with local-extremum tracking—eliminating explicit gradient computation while mitigating vanishing gradients and overfitting. We provide theoretical guarantees for EIA’s convergence and empirically validate its efficacy across multiple benchmark tasks. Results demonstrate that EIA achieves comparable or superior convergence rates and generalization performance relative to standard optimization methods. This work offers a novel perspective on the intrinsic mechanisms of neural networks and establishes a foundation for developing gradient-free, extremum-driven optimization paradigms.
📝 Abstract
We point out that neural networks are not black boxes, and their generalization stems from the ability to dynamically map a dataset to the local extrema of the model function. We further prove that the number of local extrema in a neural network is positively correlated with the number of its parameters, and on this basis, we give a new algorithm that is different from the back-propagation algorithm, which we call the extremum-increment algorithm. Some difficult situations, such as gradient vanishing and overfitting, can be reasonably explained and dealt with in this framework.