On the implicit regularization of Langevin dynamics with projected noise

2026-02-12

AI Summary

This work investigates how symmetry, through projected noise, influences the implicit regularization of stochastic gradient descent in overparameterized models. Focusing on Langevin dynamics under the action of an isometry group, the authors prove that when both the initial and target distributions are invariant under the group action, the projected-noise Langevin process is distributionally equivalent to an isotropic diffusion process augmented by an additional drift term. This drift is given by the negative gradient of the logarithm of the volume of the group orbits, which corresponds geometrically to the mean curvature of the orbits. The result is rigorously established by constructing a coupling process informed by the group structure, integrating tools from differential geometry and probability theory, and reveals a symmetry-induced mechanism underlying implicit bias in optimization.

Technology Category

• Machine Learning: Learning with Manifolds
• Search and Optimization: Non-convex Optimization
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We study Langevin dynamics with noise projected onto the directions orthogonal to an isometric group action. This mathematical model is introduced to shed new light on the effects of symmetry on stochastic gradient descent for over-parametrized models. Our main result identifies a novel form of implicit regularization: when the initial and target density are both invariant under the group action, Langevin dynamics with projected noise is equivalent in law to Langevin dynamics with isotropic diffusion but with an additional drift term proportional to the negative log volume of the group orbit. We prove this result by constructing a coupling of the two processes via a third process on the group itself, and identify the additional drift as the mean curvature of the orbits.

Problem

• implicit regularization
• Langevin dynamics
• symmetry
• group action
• over-parametrized models

Innovation

• implicit regularization
• Langevin dynamics
• projected noise
• group symmetry
• mean curvature