🤖 AI Summary
To address the non-convex optimization challenge in neural network classification—specifically, susceptibility to poor local minima and flat regions—this paper proposes SAGRAD, a batch-training algorithm integrating Simulated Annealing (SA) with Møller’s Scaled Conjugate Gradient (SCG) method. Its core innovation lies in the first incorporation of SA into the SCG framework, enabling a dynamic restart and escape mechanism that synergistically balances global exploration and local acceleration. Implemented in Fortran 77, SAGRAD incorporates efficient Hessian-vector multiplication, optimized gradient computation, and an adaptive SA weight initialization strategy. Empirical evaluation across multiple classification benchmarks demonstrates significantly improved convergence robustness and generalization performance, while markedly reducing the probability of converging to suboptimal local minima. These results validate SAGRAD’s effectiveness and practicality for non-convex optimization in neural network training.
📝 Abstract
SAGRAD (Simulated Annealing GRADient), a Fortran 77 program for computing neural networks for classification using batch learning, is discussed. Neural network training in SAGRAD is based on a combination of simulated annealing and Møller’s scaled conjugate gradient algorithm, the latter a variation of the traditional conjugate gradient method, better suited for the nonquadratic nature of neural networks. Different aspects of the implementation of the training process in SAGRAD are discussed, such as the efficient computation of gradients and multiplication of vectors by Hessian matrices that are required by Møller’s algorithm; the (re)initialization of weights with simulated annealing required to (re)start Møller’s algorithm the first time and each time thereafter that it shows insufficient progress in reaching a possibly local minimum; and the use of simulated annealing when Møller’s algorithm, after possibly making considerable progress, becomes stuck at a local minimum or flat area of weight space. Outlines of the scaled conjugate gradient algorithm, the simulated annealing procedure and the training process used in SAGRAD are presented together with results from running SAGRAD on two examples of training data.