On the Geometry and Optimization of Polynomial Convolutional Networks

2024-10-01

arXiv.org

AI Summary

This work investigates the geometric structure and optimization properties of monomial-activated convolutional neural networks (CNNs). Key open questions include the regularity and isomorphism of the parameter-to-function mapping, the dimension, degree, and singularities of the neural manifold (i.e., the set of functions representable by the network), and the number of critical points of the loss in regression tasks. Method: We establish, for the first time, an exact algebraic-geometric characterization of the mapping from parameter space to function space. Contributions: (i) We prove that the parameter mapping is almost everywhere isomorphic under filter scaling; (ii) we derive closed-form expressions for the dimension and degree of the neural manifold; (iii) we provide an explicit asymptotic formula for the number of critical points in large-scale regression. These results uncover intrinsic links between the geometry of the neural manifold and the optimization landscape, enabling quantitative analysis of expressive power versus optimization difficulty.

Technology Category

• Machine Learning: Learning with Manifolds
• Computer Vision: Learning & Optimization for CV
• Search and Optimization: Non-convex Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Social Networks and Social Media: Social media analysis through the lenses of networks
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

We study convolutional neural networks with monomial activation functions. Specifically, we prove that their parameterization map is regular and is an isomorphism almost everywhere, up to rescaling the filters. By leveraging on tools from algebraic geometry, we explore the geometric properties of the image in function space of this map - typically referred to as neuromanifold. In particular, we compute the dimension and the degree of the neuromanifold, which measure the expressivity of the model, and describe its singularities. Moreover, for a generic large dataset, we derive an explicit formula that quantifies the number of critical points arising in the optimization of a regression loss.

Problem

• Analyzes geometric properties of polynomial convolutional networks.
• Computes dimension and degree of neuromanifold for expressivity.
• Derives formula for critical points in regression loss optimization.

Innovation

• Monomial activation functions in convolutional networks
• Algebraic geometry tools analyze neuromanifold properties
• Explicit formula for critical points in regression loss