🤖 AI Summary
This work investigates how training dynamically shapes the Riemannian geometric structure induced by neural network representations. **Problem**: While deep networks operate in high-dimensional feature spaces, the geometric evolution of their induced Riemannian metric during training remains poorly understood. **Method**: We integrate Riemannian geometric analysis, infinite-width network theory, and feature-space metric modeling, conducting systematic experiments across supervised and self-supervised learning paradigms. **Contribution/Results**: We theoretically prove that infinitely wide random networks initially possess an isotropic (highly symmetric) Riemannian metric; training actively breaks this symmetry by locally amplifying the metric tensor near decision boundaries—a phenomenon we term *boundary-sensitive metric amplification*. Empirical validation across deep image classification and self-supervised learning confirms its robustness, revealing a self-emergent geometric inductive bias. This provides a novel geometric paradigm for understanding the intrinsic geometry of nonlinear feature learning.
📝 Abstract
In machine learning, there is a long history of trying to build neural networks that can learn from fewer example data by baking in strong geometric priors. However, it is not always clear a priori what geometric constraints are appropriate for a given task. Here, we explore the possibility that one can uncover useful geometric inductive biases by studying how training molds the Riemannian geometry induced by unconstrained neural network feature maps. We first show that at infinite width, neural networks with random parameters induce highly symmetric metrics on input space. This symmetry is broken by feature learning: networks trained to perform classification tasks learn to magnify local areas along decision boundaries. This holds in deep networks trained on high-dimensional image classification tasks, and even in self-supervised representation learning. These results begin to elucidate how training shapes the geometry induced by unconstrained neural network feature maps, laying the groundwork for an understanding of this richly nonlinear form of feature learning.
