🤖 AI Summary
This work investigates the asymptotic behavior of stochastic gradient descent (SGD) for permutation-invariant functions defined on high-dimensional symmetric matrices as the dimension tends to infinity. Under the setting where matrix entries are bounded and the objective function is jointly invariant under row/column permutations, the authors first establish, in the small-noise regime, that SGD trajectories converge to a deterministic gradient flow on the graphon space. Subsequently, they introduce a scaled reflected Brownian noise and derive a reflected graphon-valued stochastic differential equation (SDE) as the limiting system—marking the first extension of McKean–Vlasov theory to the graphon framework. Innovatively, they formulate the notions of *exchangeable reflected diffusion arrays* and *propagation of chaos for matrix diffusions*, and rigorously prove weak convergence of finite-dimensional SGD trajectories to the infinite-dimensional reflected graphon SDE.
📝 Abstract
We consider stochastic gradient descents on the space of large symmetric matrices of suitable functions that are invariant under permuting the rows and columns using the same permutation. We establish deterministic limits of these random curves as the dimensions of the matrices go to infinity while the entries remain bounded. Under a ``small noise'' assumption the limit is shown to be the gradient flow of functions on graphons whose existence was established in~cite{oh2021gradient}. We also consider limits of stochastic gradient descents with added properly scaled reflected Brownian noise. The limiting curve of graphons is characterized by a family of stochastic differential equations with reflections and can be thought of as an extension of the classical McKean-Vlasov limit for interacting diffusions to the graphon setting. The proofs introduce a family of infinite-dimensional exchangeable arrays of reflected diffusions and a novel notion of propagation of chaos for large matrices of diffusions converging to such arrays in a suitable sense.
