Training Neural Networks with Optimal Double-Bayesian Learning

2026-05-19

AI Summary

This work addresses the challenge of empirically setting hyperparameters such as the learning rate in neural network training, which often compromises both optimality and generalization. The authors propose a dual Bayesian learning framework that extends classical Bayesian statistics into an adversarial dual-process mechanism, enabling theoretically grounded, adaptive determination of the optimal learning rate for stochastic gradient descent. By integrating Bayesian inference, probabilistic modeling, and adversarial optimization, the method significantly enhances training stability and model performance across diverse tasks—including image classification, semantic segmentation, and object detection—thereby demonstrating the effectiveness and broad applicability of the derived learning rate schedule.

Abstract

Backpropagation with gradient descent is a common optimization strategy employed by most neural network architectures in machine learning. However, finding optimal hyperparameters to guide training has proven challenging. While it is widely acknowledged that selecting appropriate parameters is crucial for avoiding overfitting and achieving unbiased outcomes, this choice remains largely based on empirical experiments and experience. This paper presents a new probabilistic framework for the learning rate, a key parameter in stochastic gradient descent. The framework develops classic Bayesian statistics into a double-Bayesian decision mechanism involving two antagonistic Bayesian processes. A theoretically optimal learning rate can be derived from these two processes and used for stochastic gradient descent. Experiments across various classification, segmentation, and detection tasks corroborate the practical significance of the theoretically derived learning rate. The paper also discusses the ramifications of the proposed double-Bayesian framework for network training and model performance.

Problem

• learning rate
• hyperparameter optimization
• Bayesian learning
• stochastic gradient descent
• neural network training

Innovation

• double-Bayesian learning
• optimal learning rate
• stochastic gradient descent
• Bayesian optimization
• neural network training