Analysis of the Geometric Structure of Neural Networks and Neural ODEs via Morse Functions

2024-05-15

arXiv.org

AI Summary

This work investigates the geometric structure of input-output mappings for finite-depth neural networks and neural ordinary differential equations (neural ODEs), focusing on the existence, non-degeneracy, and universal embedding capacity of critical points. Using tools from Morse theory and differential topology, we develop a unified analytical framework. We prove that such mappings possess no critical points when hidden-layer dimensions are monotonically decreasing or when the phase-space dimension is at most the input dimension. Moreover, except on a Lebesgue measure-zero set of weights, critical points of bottleneck-free networks and full-rank-transform neural ODEs are non-degenerate. We further derive sufficient conditions for universal embedding. This study provides the first systematic characterization of how architectural constraints—such as dimensional transitions and rank conditions—fundamentally govern representational capacity, establishing a rigorous mathematical foundation for understanding the intrinsic dynamical mechanisms of deep learning models.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Search and Optimization: Non-convex Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

Besides classical feed-forward neural networks, also neural ordinary differential equations (neural ODEs) have gained particular interest in recent years. Neural ODEs can be interpreted as an infinite depth limit of feed-forward or residual neural networks. We study the input-output dynamics of finite and infinite depth neural networks with scalar output. In the finite depth case, the input is a state associated with a finite number of nodes, which maps under multiple non-linear transformations to the state of one output node. In analogy, a neural ODE maps an affine linear transformation of the input to an affine linear transformation of its time-$T$ map. We show that depending on the specific structure of the network, the input-output map has different properties regarding the existence and regularity of critical points, which can be characterized via Morse functions. We prove that critical points cannot exist if the dimension of the hidden layer is monotonically decreasing or the dimension of the phase space is smaller or equal to the input dimension. In the case that critical points exist, we classify their regularity depending on the specific architecture of the network. We show that except for a Lebesgue measure zero set in the weight space, each critical point is non-degenerate, if for finite depth neural networks the underlying graph has no bottleneck, and if for neural ODEs, the affine linear transformations used have full rank. For each type of architecture, the proven properties are comparable in the finite and the infinite depth case. The established theorems allow us to formulate results on universal embedding, i.e., on the exact representation of maps by neural networks and neural ODEs. Our dynamical systems viewpoint on the geometric structure of the input-output map provides a fundamental understanding of why certain architectures perform better than others.

Problem

• Analyze critical points in neural networks using Morse functions.
• Compare finite and infinite depth neural network properties.
• Investigate input-output dynamics for universal embedding capabilities.

Innovation

• Uses Morse functions for neural analysis
• Compares finite and infinite depth networks
• Classifies critical points regularity in architectures