Spectral-Refiner: Accurate Fine-Tuning of Spatiotemporal Fourier Neural Operator for Turbulent Flows

2024-05-27

AI Summary

Fourier Neural Operators (FNOs) suffer from high training costs, limited accuracy, and inability to support arbitrary temporal super-resolution in turbulent flow simulation. Method: We propose a spatiotemporal spectral fine-tuning framework featuring (1) a novel Bochner-space-mapped spatiotemporal adaptive FNO; (2) truncation-free spectral convolution layers enabling efficient fine-tuning; and (3) integration of the negative Sobolev norm and a rigorous a posteriori error estimator—derived from Parseval’s identity—into the loss function to ensure convexity and physical consistency. Results: Evaluated on Navier–Stokes equation benchmarks, our framework significantly outperforms end-to-end FNO training and, in specific regimes, surpasses conventional numerical solvers—achieving superior accuracy, computational efficiency, and generalizability across diverse flow configurations.

Technology Category

• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Neuro-Symbolic Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Recent advancements in operator-type neural networks have shown promising results in approximating the solutions of spatiotemporal Partial Differential Equations (PDEs). However, these neural networks often entail considerable training expenses, and may not always achieve the desired accuracy required in many scientific and engineering disciplines. In this paper, we propose a new learning framework to address these issues. A new spatiotemporal adaptation is proposed to generalize any Fourier Neural Operator (FNO) variant to learn maps between Bochner spaces, which can perform an arbitrary-length temporal super-resolution for the first time. To better exploit this capacity, a new paradigm is proposed to refine the commonly adopted end-to-end neural operator training and evaluations with the help from the wisdom from traditional numerical PDE theory and techniques. Specifically, in the learning problems for the turbulent flow modeled by the Navier-Stokes Equations (NSE), the proposed paradigm trains an FNO only for a few epochs. Then, only the newly proposed spatiotemporal spectral convolution layer is fine-tuned without the frequency truncation. The spectral fine-tuning loss function uses a negative Sobolev norm for the first time in operator learning, defined through a reliable functional-type a posteriori error estimator whose evaluation is exact thanks to the Parseval identity. Moreover, unlike the difficult nonconvex optimization problems in the end-to-end training, this fine-tuning loss is convex. Numerical experiments on commonly used NSE benchmarks demonstrate significant improvements in both computational efficiency and accuracy, compared to end-to-end evaluation and traditional numerical PDE solvers under certain conditions. The source code is publicly available at https://github.com/scaomath/torch-cfd.

Problem

• Fine-tune spatiotemporal Fourier Neural Operator
• Enhance turbulent flow modeling accuracy
• Reduce training costs in neural networks

Innovation

• Spectral convolution layer fine-tuning
• Negative Sobolev norm loss function
• Functional-type a posteriori error estimator