Adaptive Norm-Based Regularization for Neural Networks

2026-04-30

AI Summary

This study addresses the limited generalization performance of standard norm-based regularization in neural networks when dealing with high-dimensional or feature-correlated settings, where conventional methods inadequately control model complexity. To overcome this, the authors propose two covariance-aware adaptive regularization techniques: first, incorporating the input feature covariance structure into ℓ₂ weight decay to refine ridge-type penalties; second, combining ℓ₁ sparsity with covariance-informed ℓ₂ regularization to achieve structured sparsity. By innovatively embedding feature covariance information into classical Lasso and ridge regression frameworks, the approach enables more precise complexity control. Extensive experiments on Monte Carlo simulations and real-world datasets—including building cooling load prediction and leukemia cell classification—demonstrate that the proposed methods significantly outperform traditional regularization strategies and substantially enhance model generalization.

Abstract

In this paper, we study norm-based regularization methods for neural networks. We compare existing penalization approaches and introduce two regularization strategies that extend classical ridge- and lasso-type penalties to neural network models. The first strategy modifies weight decay by incorporating the covariance structure of the input features into a ridge-type $\ell_2$ penalty, allowing regularization to account for feature dependence. The second combines an $\ell_1$ sparsity penalty with covariance-aware $\ell_2$ regularization, producing neural network weights that are both sparse and structurally informed. Monte Carlo simulations are used to evaluate these methods under different data-generating settings, followed by two real-data applications on building cooling-load prediction and leukemia cell-type classification from high-dimensional gene expression data. Across simulated and real-data examples, the proposed regularizers improve predictive performance on unseen data and provide more effective complexity control than standard norm-based penalties, particularly when features are correlated or high-dimensional.

Problem

• norm-based regularization
• neural networks
• feature correlation
• high-dimensional data
• model complexity

Innovation

• covariance-aware regularization
• adaptive norm-based regularization
• structured sparsity
• neural network regularization
• feature dependence