Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation

2024-06-04

arXiv.org

AI Summary

To address the challenge of efficiently and accurately approximating high-dimensional smooth and nonsmooth functions as well as partial differential equations (PDEs), this paper proposes Polynomial-Augmented Neural Networks (PANNs), which seamlessly integrate deep neural networks with orthogonal polynomial approximation. Our method introduces (1) a weak orthogonality constraint enforcing mutual orthogonality between network hidden layers and polynomial bases, and (2) a joint framework combining basis pruning and polynomial preconditioning to enhance numerical stability, accuracy, and scalability to high dimensions. Experiments demonstrate that PANNs consistently outperform pure deep neural networks and traditional polynomial methods on both function regression and PDE solving tasks. Notably, they achieve substantial accuracy gains for low-regularity functions, while exhibiting rapid convergence and strong generalization capability.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Non-convex Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Social Networks and Social Media: Computational social science

Abstract

We present polynomial-augmented neural networks (PANNs), a novel machine learning architecture that combines deep neural networks (DNNs) with a polynomial approximant. PANNs combine the strengths of DNNs (flexibility and efficiency in higher-dimensional approximation) with those of polynomial approximation (rapid convergence rates for smooth functions). To aid in both stable training and enhanced accuracy over a variety of problems, we present (1) a family of orthogonality constraints that impose mutual orthogonality between the polynomial and the DNN within a PANN; (2) a simple basis pruning approach to combat the curse of dimensionality introduced by the polynomial component; and (3) an adaptation of a polynomial preconditioning strategy to both DNNs and polynomials. We test the resulting architecture for its polynomial reproduction properties, ability to approximate both smooth functions and functions of limited smoothness, and as a method for the solution of partial differential equations (PDEs). Through these experiments, we demonstrate that PANNs offer superior approximation properties to DNNs for both regression and the numerical solution of PDEs, while also offering enhanced accuracy over both polynomial and DNN-based regression (each) when regressing functions with limited smoothness.

Problem

• Enhancing function and PDE approximation
• Combining neural networks with polynomial approximants
• Introducing weak orthogonality constraints for stability

Innovation

• Polynomial-augmented neural networks
• Weak orthogonality constraints
• Basis pruning for dimensionality