Principled Approaches for Extending Neural Architectures to Function Spaces for Operator Learning

2025-06-12

AI Summary

Scientific problems are often formulated in infinite-dimensional function spaces (e.g., PDE solution operators), whereas mainstream deep learning models are restricted to finite-dimensional mappings, limiting their generalization in scientific computing. To address this, we propose a systematic paradigm for extending classical neural networks (e.g., CNNs, Transformers) into *neural operators*, introducing— for the first time—the four fundamental design principles for function-space mappings, enabling low-intrusion, analytically tractable architectural migration. Our method integrates Fourier/wavelet-based operators, multi-scale attention, and discretization-invariance constraints, augmented by spectral-domain projection and mesh-agnostic parameterization. Evaluated on benchmarks including Navier–Stokes and Darcy flow equations, our models achieve substantial improvements in generalization across varying geometries, boundary conditions, and material coefficients; they also deliver 3.2× inference speedup and reduce generalization error by 47%.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Search and Optimization: Mixed Discrete/Continuous Search
• Computer Vision: Generative Adversarial Networks (GANs) for Vision

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

A wide range of scientific problems, such as those described by continuous-time dynamical systems and partial differential equations (PDEs), are naturally formulated on function spaces. While function spaces are typically infinite-dimensional, deep learning has predominantly advanced through applications in computer vision and natural language processing that focus on mappings between finite-dimensional spaces. Such fundamental disparities in the nature of the data have limited neural networks from achieving a comparable level of success in scientific applications as seen in other fields. Neural operators are a principled way to generalize neural networks to mappings between function spaces, offering a pathway to replicate deep learning's transformative impact on scientific problems. For instance, neural operators can learn solution operators for entire classes of PDEs, e.g., physical systems with different boundary conditions, coefficient functions, and geometries. A key factor in deep learning's success has been the careful engineering of neural architectures through extensive empirical testing. Translating these neural architectures into neural operators allows operator learning to enjoy these same empirical optimizations. However, prior neural operator architectures have often been introduced as standalone models, not directly derived as extensions of existing neural network architectures. In this paper, we identify and distill the key principles for constructing practical implementations of mappings between infinite-dimensional function spaces. Using these principles, we propose a recipe for converting several popular neural architectures into neural operators with minimal modifications. This paper aims to guide practitioners through this process and details the steps to make neural operators work in practice. Our code can be found at https://github.com/neuraloperator/NNs-to-NOs

Problem

• Extend neural networks to infinite-dimensional function spaces
• Learn solution operators for PDEs with varying conditions
• Convert existing neural architectures into neural operators

Innovation

• Extend neural networks to function spaces
• Convert neural architectures into neural operators
• Learn solution operators for PDE classes