🤖 AI Summary
This work addresses the practical limitations of neural networks stemming from insufficient generalization and the lack of systematic evaluation of existing regularization techniques. To this end, the authors propose a unified categorization framework encompassing four major dimensions—data, architecture, training, and loss functions—and conduct a comprehensive survey coupled with empirical analysis of mainstream regularization methods. Large-scale comparative experiments are performed using multilayer perceptrons and convolutional neural networks across ten datasets spanning numerical and image classification tasks. The findings reveal that regularization efficacy is highly data-dependent: for instance, traditional regularization terms are effective primarily on numerical data, whereas batch normalization predominantly enhances performance on image-related tasks. This study thus provides both theoretical insights and practical guidance for the selection and design of regularization strategies.
📝 Abstract
Despite huge successes on a wide range of tasks, neural networks are known to sometimes struggle to generalise to unseen data. Many approaches have been proposed over the years to promote the generalisation ability of neural networks, collectively known as regularisation techniques. These are used as common practice under the assumption that any regularisation added to the pipeline would result in a performance improvement. In this study, we investigate whether this assumption holds in practice. First, we provide a broad review of regularisation techniques, including modern theories such as double descent. We propose a taxonomy of methods under four broad categories, namely: (1) data-based strategies, (2) architecture strategies, (3) training strategies, and (4) loss function strategies. Notably, we highlight the contradictions and correspondences between the approaches in these broad classes. Further, we perform an empirical comparison of the various regularisation techniques on classification tasks for ten numerical and image datasets applied to the multi-layer perceptron and convolutional neural network architectures. Results show that the efficacy of regularisation is dataset-dependent. For example, the use of a regularisation term only improved performance on numeric datasets, whereas batch normalisation improved performance on image datasets only. Generalisation is crucial to machine learning; thus, understanding the effects of applying regularisation techniques, and considering the connections between them is essential to the appropriate use of these methods in practice.