On the Geometry of Deep Learning

2024-08-09

arXiv.org

AI Summary

This work investigates the geometric nature of deep neural networks (DNNs) in solving high-dimensional linear problems: specifically, how stacked nonlinear activations (e.g., ReLU) induce piecewise-linear mappings via polyhedral tessellations of the input space. Methodologically, we integrate convex analysis, piecewise-linear function theory, computational geometry, and interpretability modeling to rigorously characterize the cumulative effect of each layer’s transformation on spatial partitioning. Our core contribution is the first provably sound geometric representation framework that establishes a strict equivalence between DNN mappings and multidimensional affine splines—thereby unifying explanations of expressive capacity, generalization bounds, and robustness mechanisms. This framework provides foundational theoretical support for model compression, training optimization, and trustworthy AI.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Computer Vision: Representation Learning for Vision

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Web Mining and Content Analysis: Models for Web evolution

Abstract

In this paper, we overview one promising avenue of progress at the mathematical foundation of deep learning: the connection between deep networks and function approximation by affine splines (continuous piecewise linear functions in multiple dimensions). In particular, we will overview work over the past decade on understanding certain geometrical properties of a deep network's affine spline mapping, in particular how it tessellates its input space. As we will see, the affine spline connection and geometrical viewpoint provide a powerful portal through which to view, analyze, and improve the inner workings of a deep network.

Problem

• Deep Learning Models
• Multidimensional Spatial Linear Problems
• Shape Attributes for Space Partitioning

Innovation

• deep learning
• affine splines
• geometric approaches