A Relative Homology Theory of Representation in Neural Networks

2025-02-03

AI Summary

This paper addresses the topological characterization of representations in ReLU neural networks. Method: We propose a metric-free analytical framework based on relative homology. Leveraging the piecewise affine structure induced by ReLU networks and the convexity of its intersection with the input manifold, we establish an isomorphism between the representation’s homology groups and the relative homology groups $H_k(M, O_Phi)$, where the overlap decomposition $O_Phi$ encodes the coverage and intersection pattern of linear regions in the input space. Contribution/Results: We are the first to formalize neural representation analysis as a relative homology problem; we rigorously prove that $O_Phi$ fully determines the homological invariants; and we design a metric-free algorithm for computing Betti numbers via linear programming and union-find. Experiments on synthetic data validate its ability to isolate pure topological features, reveal evolutionary patterns of $O_Phi$ during training, and quantify how network width and depth affect topological expressivity.

Technology Category

• Machine Learning: Representation Learning
• Computer Vision: Representation Learning for Vision
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Web Mining and Content Analysis: Models for Web evolution

Abstract

Previous research has proven that the set of maps implemented by neural networks with a ReLU activation function is identical to the set of piecewise linear continuous maps. Furthermore, such networks induce a hyperplane arrangement splitting the input domain into convex polyhedra $G_J$ over which the network $Phi$ operates in an affine manner. In this work, we leverage these properties to define the equivalence class of inputs $sim_Phi$, which can be split into two sets related to the local rank of $Phi_J$ and the intersections $cap ext{Im}Phi_{J_i}$. We refer to the latter as the overlap decomposition $O_Phi$ and prove that if the intersections between each polyhedron and the input manifold are convex, the homology groups of neural representations are isomorphic to relative homology groups $H_k(Phi(M)) simeq H_k(M,O_Phi)$. This lets us compute Betti numbers without the choice of an external metric. We develop methods to numerically compute the overlap decomposition through linear programming and a union-find algorithm. Using this framework, we perform several experiments on toy datasets showing that, compared to standard persistent homology, our relative homology-based computation of Betti numbers tracks purely topological rather than geometric features. Finally, we study the evolution of the overlap decomposition during training on various classification problems while varying network width and depth and discuss some shortcomings of our method.

Problem

• Neural Networks
• ReLU Activation Function
• Input Representation

Innovation

• ReLU Neural Networks
• Betti Numbers
• Overlap Analysis