Activation degree thresholds and expressiveness of polynomial neural networks

2024-08-08

AI Summary

This work investigates the expressive power of polynomial neural networks, focusing on the “activation threshold”—a geometric invariant defined by the neural cluster structure, i.e., the critical activation strength at which neural diversity dimension first attains its theoretical maximum. We formally define this novel concept and, for the first time, prove the “high-activation conjecture.” We establish the universal existence of an activation threshold for all width-unconstrained polynomial networks and derive a quadratic upper bound dependent on the network’s maximum width. Notably, for equi-width architectures, the threshold is exactly 1, at which the neural cluster dimension achieves the theoretical upper bound—revealing an exact correspondence between architectural symmetry and expressivity. Our methodology integrates tools from algebraic geometry, polynomial mapping theory, and dimension theory. Key contributions include: (i) proving the universal existence of activation thresholds; (ii) providing a computable quadratic upper bound; and (iii) characterizing structural optimality via threshold minimization.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Web Mining and Content Analysis: Models for Web evolution

Abstract

We study the expressive power of deep polynomial neural networks through the geometry of their neurovariety. We introduce the notion of the activation degree threshold of a network architecture to express when the dimension of the neurovariety achieves its theoretical maximum. We prove the existence of the activation degree threshold for all polynomial neural networks without width-one bottlenecks and demonstrate a universal upper bound that is quadratic in the width of largest size. In doing so, we prove the high activation degree conjecture of Kileel, Trager, and Bruna. Certain structured architectures have exceptional activation degree thresholds, making them especially expressive in the sense of their neurovariety dimension. In this direction, we prove that polynomial neural networks with equi-width architectures are maximally expressive by showing their activation degree threshold is one.

Problem

• Expressive power of polynomial neural networks
• Activation degree threshold determination
• Maximal expressiveness in equi-width architectures

Innovation

• Introduces activation degree threshold concept
• Proves existence of activation degree threshold
• Shows equi-width architectures are maximally expressive