Why can genetic algorithms work in high-dimensional search spaces?

2026-06-29

AI Summary

This study investigates the intrinsic mechanisms underlying the efficiency of genetic algorithms in high-dimensional spaces. By analyzing the dynamics of the elitist (1+M) genetic algorithm in the small-mutation limit, the work establishes—for the first time—a theoretical equivalence between the algorithm and a truncated gradient descent process driven by anisotropic Gaussian white noise. This equivalence demonstrates that the algorithm implicitly follows the gradient direction of the loss function without explicitly computing or averaging gradients. Integrating tools from stochastic processes, Hessian spectral analysis, and high-dimensional optimization theory, the paper reveals that the algorithm’s efficiency depends on the effective rank of the loss function’s Hessian matrix rather than the total number of parameters. In high-dimensional problems such as neural networks, the Hessian spectrum is highly concentrated, rendering the effective rank much smaller than the ambient parameter dimension, thereby suppressing noise and explaining the scalability of genetic algorithms.

Abstract

We show that the effective dynamics of the elitist $(1+M)$ genetic algorithm is, in the limit of small mutations, clipped gradient descent on the loss in the presence of anisotropic Gaussian white noise. In expectation, therefore, a simple mutation-selection genetic algorithm follows the gradient of the loss, without explicit calculation of gradients and without averaging over loss evaluations. The genetic algorithm is slower than gradient descent because of the noise that acts in directions transverse to the gradient. However, this slowdown is controlled not by the number of parameters of the search space but by the effective rank of the Hessian of the loss function. For the concentrated Hessian spectra observed in neural-network loss functions the effective rank can be far smaller than the number of parameters, which may explain why genetic algorithms can scale to large search spaces.

Problem

• genetic algorithms
• high-dimensional search spaces
• gradient descent
• Hessian spectrum
• effective rank

Innovation

• genetic algorithms
• gradient descent
• anisotropic noise
• Hessian effective rank
• high-dimensional optimization