Training Deep Neural Classifiers with Soft Diamond Regularizers

2024-12-30

AI Summary

Balancing sparsity and generalization in neural network classifiers remains challenging. To address this, we propose the Soft Diamond Regularizer, the first method to adopt heavy-tailed α-stable distributions as weight priors, integrated with a tunable soft-constraint mechanism that enhances synaptic sparsity without compromising accuracy. Our approach accelerates α-stable density evaluation via precomputed lookup tables and seamlessly integrates into standard deep learning training pipelines. Experiments on CIFAR-10, CIFAR-100, and Caltech-256 demonstrate classification accuracy improvements of 4.57%, 4.27%, and 6.69%, respectively, over L₁, L₂, and hard diamond regularizers. The core contribution is the design of the first differentiable, tunable soft diamond constraint grounded in α-stable distributions—overcoming expressivity limitations and optimization bottlenecks inherent in conventional regularization techniques.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Reasoning under Uncertainty: Stochastic Optimization
• Computer Vision: Learning & Optimization for CV

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Security and Privacy: Large-scale security measurements

Abstract

We introduce new emph{soft diamond} regularizers that both improve synaptic sparsity and maintain classification accuracy in deep neural networks. These parametrized regularizers outperform the state-of-the-art hard-diamond Laplacian regularizer of Lasso regression and classification. They use thick-tailed symmetric alpha-stable ($mathcal{S alpha S}$) bell-curve synaptic weight priors that are not Gaussian and so have thicker tails. The geometry of the diamond-shaped constraint set varies from a circle to a star depending on the tail thickness and dispersion of the prior probability density function. Training directly with these priors is computationally intensive because almost all $mathcal{S alpha S}$ probability densities lack a closed form. A precomputed look-up table removed this computational bottleneck. We tested the new soft diamond regularizers with deep neural classifiers on the three datasets CIFAR-10, CIFAR-100, and Caltech-256. The regularizers improved the accuracy of the classifiers. The improvements included $4.57%$ on CIFAR-10, $4.27%$ on CIFAR-100, and $6.69%$ on Caltech-256. They also outperformed $L_2$ regularizers on all the test cases. Soft diamond regularizers also outperformed $L_1$ lasso or Laplace regularizers because they better increased sparsity while improving classification accuracy. Soft-diamond priors substantially improved accuracy on CIFAR-10 when combined with dropout, batch, or data-augmentation regularization.

Problem

• Neural Network Classifier
• Accuracy Improvement
• Information Connectivity

Innovation

• Soft Diamond Method
• Neural Network Optimization
• Classification Accuracy Enhancement