Deep Neural Networks with General Activations: Super-Convergence in Sobolev Norms

2025-08-07

AI Summary

This work addresses the lack of rigorous error theory for neural-network-based solvers of partial differential equations (PDEs). We systematically investigate the approximation capacity of deep fully connected networks for weak solutions of PDEs in the Sobolev space $W^{n,infty}$. Under general activation functions—including common non-polynomial ones—we derive high-order approximation error estimates in the $W^{m,p}$ norm for $m < n$ and $1 leq p leq infty$. Crucially, we identify and rigorously prove a “superconvergence” phenomenon: deep networks can achieve convergence rates exceeding the intrinsic limits of classical finite element and spectral methods. This result fills a fundamental theoretical gap in the error analysis of deep learning for scientific computing. By establishing a unified, scalable Sobolev framework, our work provides a solid theoretical foundation for both the reliability and efficiency of neural PDE solvers.

Technology Category

• Machine Learning: Deep Learning Theory
• Natural Language Processing: Learning & Optimization for NLP
• Computer Vision: Learning & Optimization for CV

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Social Networks and Social Media: Influence propagation, information diffusion, and the prediction on networks
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

This paper establishes a comprehensive approximation result for deep fully-connected neural networks with commonly-used and general activation functions in Sobolev spaces $W^{n,infty}$, with errors measured in the $W^{m,p}$-norm for $m < n$ and $1le p le infty$. The derived rates surpass those of classical numerical approximation techniques, such as finite element and spectral methods, exhibiting a phenomenon we refer to as emph{super-convergence}. Our analysis shows that deep networks with general activations can approximate weak solutions of partial differential equations (PDEs) with superior accuracy compared to traditional numerical methods at the approximation level. Furthermore, this work closes a significant gap in the error-estimation theory for neural-network-based approaches to PDEs, offering a unified theoretical foundation for their use in scientific computing.

Problem

• Approximating deep neural networks in Sobolev spaces
• Surpassing classical numerical methods in PDE solutions
• Closing error-estimation gaps for neural-network PDE approaches

Innovation

• Deep networks with general activation functions
• Super-convergence in Sobolev norms
• Superior accuracy for PDE weak solutions