Singular Learning and Occam's Razor in Deep Monomial Networks

2026-06-26

AI Summary

This work investigates how architecture-induced critical points in deep neural network optimization steer convergence toward simpler functions, thereby uncovering the mathematical origins of implicit bias. Focusing on deep monomial networks, the study pioneers a synthesis of singular learning theory and Occam’s razor, integrating tools from polynomial algebra—such as Mason’s theorem—and Jacobian rank-deficiency analysis to rigorously characterize the algebraic-geometric structure of singular critical points. The analysis demonstrates that, when the number of activations is sufficiently large, these critical points correspond precisely to subnetwork configurations involving neuron failure or redundancy. This correspondence provides a novel theoretical foundation for understanding implicit regularization and generalization in deep networks.

Abstract

In the optimization of neural networks, gradient dynamics are influenced by critical points that arise from the model's architecture. These critical points occur where the Jacobian of the model's parametrization is rank-deficient, and are the most pronounced singularities studied in Singular Learning Theory. We investigate such points in deep fully-connected networks with monomial activations via tools from polynomial algebra such as Mason's Theorem. We show that, for sufficiently large activation degree, criticality occurs precisely at subnetworks, i.e., at parameter configurations where some neurons are inactive or redundant. This offers a mathematical perspective on the implicit bias in deep neural networks, explaining the tendency of these models to converge toward simpler functions.

Problem

• Singular Learning
• Deep Monomial Networks
• Critical Points
• Implicit Bias
• Occam's Razor

Innovation

• Singular Learning Theory
• monomial activation
• implicit bias
• critical points
• Occam's Razor