Low-dimensional topology of deep neural networks

2026-06-30

AI Summary

This study investigates the capacity of deep neural networks to transform topological structures—such as linking numbers—within low-dimensional representation spaces, introducing low-dimensional topological invariants as a novel measure of expressive power. By constraining network layers to ℝ³ and combining theoretical analysis with visual experiments, the authors systematically compare the topological expressivity of feedforward networks, ResNets, and Transformers. They find that ResNets and Transformers exhibit comparable and significantly superior performance in altering linking numbers relative to feedforward networks employing monotonic activation functions; however, incorporating non-monotonic activations enables feedforward architectures to match this capability. These findings generalize to any dimension d > 3, revealing an equivalence between skip connections and attention mechanisms in facilitating topological transformations and underscoring the critical role of non-monotonic activations.

Abstract

We study layered models, including feedforward networks, ResNets, and transformers, by limiting each layer to a width of $d = 3$, i.e., $\mathbb{R}^3$ as representation space. This allows us to track how a neural network changes low-dimensional topological invariants through its layers. Just about any topological structure may be simplified or even trivialized by simply increasing dimension; e.g., any knot is equivalent to an unknot in $\mathbb{R}^4$. By restricting to $\mathbb{R}^3$, we not only isolate the effects of activation and depth from that of width, we work in a space that lends itself to easy visualization. We focus on linking number here, deferring other invariants like link groups, Milnor's $\barμ$-invariants, knot types, ambient cobordisms, to a sequel. We provide full proofs and empirical experiments to justify the following insights: When measured by their power to effect changes in linking numbers, the layer-skipping feature in ResNets is as powerful as the attention mechanism in transformers; both ResNets and transformers are strictly more powerful than feedforward neural networks with monotonic activations, which are in turn more powerful than invertible and flow-based models; but replacing monotonic activation with a nonmonotonic one elevates a feedforward network into the same expressivity class as ResNets and transformers. These results suggest that low-dimensional topology can be a useful tool to guide designs of AI architectures. We also generalize our results from $d = 3$ to arbitrary $d > 3$.

Problem

• low-dimensional topology
• neural network expressivity
• linking number
• representation space
• topological invariants

Innovation

• low-dimensional topology
• linking number
• neural network expressivity
• ResNet
• transformer