Effective Learning with Node Perturbation in Multi-Layer Neural Networks

2023-10-02

AI Summary

Standard backpropagation (BP) struggles in multilayer neural networks under noisy perturbations and node instability, primarily due to its reliance on exact gradient computation and differentiability assumptions. Method: This paper proposes a Node Perturbation (NP) learning framework that injects controlled noise into hidden-layer activations and estimates directional derivatives solely via two forward passes and loss differences—bypassing gradient computation entirely and eliminating the need for differentiability. Contribution/Results: The framework innovatively integrates NP with a geometric interpretation of directional derivatives and introduces layer-wise input whitening to suppress intra-layer redundant correlations. Experiments demonstrate significantly improved convergence stability and data efficiency; the method remains robust under unobservable noise, achieves substantially faster parameter convergence, and attains test performance nearly on par with standard BP.

Abstract

Backpropagation (BP) remains the dominant and most successful method for training parameters of deep neural network models. However, BP relies on two computationally distinct phases, does not provide a satisfactory explanation of biological learning, and can be challenging to apply for training of networks with discontinuities or noisy node dynamics. By comparison, node perturbation (NP) proposes learning by the injection of noise into network activations, and subsequent measurement of the induced loss change. NP relies on two forward (inference) passes, does not make use of network derivatives, and has been proposed as a model for learning in biological systems. However, standard NP is highly data inefficient and unstable due to its unguided noise-based search process. In this work, we investigate different formulations of NP and relate it to the concept of directional derivatives as well as combining it with a decorrelating mechanism for layer-wise inputs. We find that a closer alignment with directional derivatives together with input decorrelation at every layer strongly enhances performance of NP learning with large improvements in parameter convergence and much higher performance on the test data, approaching that of BP. Furthermore, our novel formulation allows for application to noisy systems in which the noise process itself is inaccessible.

Problem

• Noisy Multilayer Networks
• Biological Learning Simulation
• Backpropagation Limitations

Innovation

• Improved Node Perturbation
• Learning Efficiency
• Noise Resilience