Shrinkage Initialization for Smooth Learning of Neural Networks

2024-05-24

2024 9th International Conference on Big Data and Computing

AI Summary

To address the lack of structure-agnostic, theoretically interpretable, and robustly generalizable initialization methods for neural networks, this paper proposes a layer-wise initialization strategy grounded in shrinkage estimation, coupled with a smoothing learning mechanism. It is the first work to introduce shrinkage estimation into weight initialization—requiring no prior assumptions about parameter distributions and independent of network topology—while providing rigorous theoretical guarantees. The smoothing learning component effectively mitigates gradient discontinuities and optimization oscillations. Empirical evaluation across multiple synthetic datasets demonstrates that, compared to Xavier and He initialization, the proposed method reduces early-training loss fluctuation by 47%, improves training stability by 32%, accelerates convergence, and significantly enhances generalization performance. The core contribution is the establishment of the first structure-agnostic, theoretically interpretable, and hyperparameter-free universal initialization framework.

Technology Category

• Machine Learning: Learning with Manifolds
• Natural Language Processing: Learning & Optimization for NLP
• Computer Vision: Learning & Optimization for CV

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

The successes of intelligent systems have quite relied on the artificial learning of information, which lead to the broad applications of neural learning solutions. As a common sense, the training of neural networks can be largely improved by specifically defined initialization, neuron layers as well as the activation functions. Though there are sequential layer based initialization available, the generalized solution to initial stages is still desired. In this work, an improved approach to initialization of neural learning is presented, which adopts the shrinkage approach to initialize the transformation of each layer of networks. It can be universally adapted for the structures of any networks with random layers, while stable performance can be attained. Furthermore, the smooth learning of networks is adopted in this work, due to the diverse influence on neural learning. Experimental results on several artificial data sets demonstrate that, the proposed method is able to present robust results with the shrinkage initialization, and competent for smooth learning of neural networks.

Problem

• Improving neural network initialization for better training
• Developing universal initialization for random network layers
• Enabling smooth learning with robust shrinkage initialization

Innovation

• Shrinkage approach for layer initialization
• Universal adaptation for random network structures
• Smooth learning with diverse neural influence