Beyond Backpropagation: Monte Carlo Method Can Train Deep Neural Networks

2026-07-09

AI Summary

This work addresses the challenges of vanishing and exploding gradients in backpropagation by proposing a gradient-free training method for deep neural networks. Built upon an extremely simple Monte Carlo strategy, the approach randomly perturbs network parameters and retains updates that reduce the loss, enabling effective optimization on a single GPU. The study demonstrates, for the first time, that such a minimalist gradient-free algorithm can directly train networks exceeding 20 layers without relying on batch normalization or residual connections. Moreover, it is compatible with purely pruned architectures, discrete weights, and non-standard activation functions such as Gaussian activations. Experiments validate the method’s efficacy and generality by successfully training ultra-deep networks, wide networks with 16,384 neurons, and a minimal Transformer on MNIST and Tiny Shakespeare benchmarks.

Abstract

Backpropagation (BP) dominates deep learning training, but its reliance on gradients brings inherent troubles -- vanishing and exploding gradients. The pursuit of gradient-free methods has long been a goal in the field of artificial intelligence. This paper shows that indeed the simplest Monte Carlo algorithm implemented on a single GPU -- randomly mutate a parameter, keep it if the loss decreases, otherwise retry -- can practically train deep networks. This gradient-free method does not even need common techniques such as batch normalization or residual connections to directly train sufficiently deep networks. More remarkably, its flexibility extends to several nontrivial scenarios: it enables pure pruning training, supports discrete weights, accommodates unconventional transfer functions such as Gaussian, and reveals the substantial redundancy of deep networks. We have demonstrated its feasibility on deep networks with more than 20 layers, single-hidden-layer wide networks with up to 16,384 hidden neurons, and even a simple Transformer architecture trained on both image classification (MNIST) and character-level language modeling (Tiny Shakespeare). This simple gradient-free method may offer a complementary perspective for understanding the self-organization and learning mechanisms of neural networks, and also provides an alternative route for building physically inspired deep learning systems.

Problem

• backpropagation
• gradient-free
• deep neural networks
• vanishing gradients
• exploding gradients

Innovation

• gradient-free training
• Monte Carlo method
• deep neural networks
• parameter mutation
• network redundancy