Langevin dynamics along the zero set of real-analytic potentials

2026-08-10

AI Summary

This work investigates why stochastic optimization in high-dimensional over-parameterized models favors singular solutions with good generalization. Focusing on the limiting dynamics of Langevin processes constrained to the zero set of a non-negative real-analytic potential as the inverse temperature β tends to infinity, we introduce the local learning coefficient into the Langevin framework for the first time. By stratifying the zero set according to the values and multiplicities of this coefficient, we derive an associated hierarchical stochastic process. Leveraging tools from real-analytic geometry, singularity theory, and Watanabe’s singular learning theory, we prove that the corresponding Dirichlet forms converge to hierarchical Dirichlet forms supported on each stratum. This reveals a dynamical mechanism driving the system toward higher-dimensional singular submanifolds, offering a theoretical explanation for the empirical observation that SGD in deep learning preferentially converges to well-generalizing solutions.

Abstract

We consider the Langevin diffusion $dX_t = - β\nabla V(X_t) dt + \sqrt{2} dB_t$ for a general nonnegative real-analytic potential $V$ and a large parameter $β$. In the large-$β$ limit the process is confined to the zero set of $V$, assuming that it starts there. We derive a candidate limiting evolution on the zero set. To do so, the zero set is partitioned into strata according to a measure of local codimension known as the local learning coefficient and its multiplicity. It is then shown that the Dirichlet form associated with $X$ converges in a certain sense to a hierarchy of Dirichlet forms corresponding to a stochastic evolution on the strata. This evolution is strongly biased toward higher-dimensional, or "more singular", strata. This result is motivated by a question from Watanabe's singular learning theory regarding the learning dynamics of overparameterized statistical models and the generalization puzzle in deep learning. The result suggests a mechanism for the observation that stochastic gradient methods tend to be biased toward singular solutions that generalize well.

Problem

• Langevin dynamics
• real-analytic potentials
• zero set
• singular learning theory
• generalization puzzle

Innovation

• Langevin dynamics
• real-analytic potentials
• zero set stratification
• local learning coefficient
• Dirichlet form convergence