On Space Folds of ReLU Neural Networks

2025-02-14

AI Summary

This work investigates the “spatial folding” phenomenon in ReLU neural networks—specifically, how straight-line paths in Euclidean input space map to the Hamming activation space and why their convexity is lost. Method: The authors introduce the first range-based metric framework for quantifying folding, rigorously proving the equivalence between convexity preservation in input space and Hamming activation space. They employ geometric modeling and self-similarity analysis to characterize the folding process. Contribution/Results: The analysis reveals that spatial folding exhibits fractal structure; experiments on CantorNet and MNIST validate the metric’s effectiveness, computational tractability, and self-similarity. Collectively, this work establishes a novel paradigm for understanding the information-geometric nature of ReLU networks and provides an interpretable, quantitative tool for analyzing activation-space transformations.

Technology Category

• Machine Learning: Learning with Manifolds
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Web Mining and Content Analysis: Models for Web evolution
• Social Networks and Social Media: Influence propagation, information diffusion, and the prediction on networks

Abstract

Recent findings suggest that the consecutive layers of ReLU neural networks can be understood geometrically as space folding transformations of the input space, revealing patterns of self-similarity. In this paper, we present the first quantitative analysis of this space folding phenomenon in ReLU neural networks. Our approach focuses on examining how straight paths in the Euclidean input space are mapped to their counterparts in the Hamming activation space. In this process, the convexity of straight lines is generally lost, giving rise to non-convex folding behavior. To quantify this effect, we introduce a novel measure based on range metrics, similar to those used in the study of random walks, and provide the proof for the equivalence of convexity notions between the input and activation spaces. Furthermore, we provide empirical analysis on a geometrical analysis benchmark (CantorNet) as well as an image classification benchmark (MNIST). Our work advances the understanding of the activation space in ReLU neural networks by leveraging the phenomena of geometric folding, providing valuable insights on how these models process input information.

Problem

• Quantify ReLU network space folding
• Map Euclidean to Hamming space paths
• Analyze non-convex folding behavior

Innovation

• Quantitative analysis of space folding
• Novel range metrics for folding behavior
• Empirical analysis on geometric benchmarks