Efficient Transformer-Inspired Variants of Physics-Informed Deep Operator Networks

2025-09-01

AI Summary

To address the trade-off between accuracy and efficiency in DeepONets for solving partial differential equations (PDEs), this paper proposes a novel Transformer-inspired DeepONet variant. The core innovation is a non-intrusive bidirectional cross-attention mechanism that enables dynamic, symmetric fusion of query-point and input-function information between the branch and trunk networks. This design preserves the original DeepONet’s architectural simplicity and training efficiency while substantially enhancing representational capacity and naturally aligning the optimal variant with the underlying PDE physics. Evaluated on four benchmark PDEs—advection, diffusion-reaction, Burgers, and KdV equations—the method matches or exceeds the accuracy of state-of-the-art DeepONet variants while significantly accelerating training. Robustness and generalizability are rigorously validated via Wilcoxon signed-rank tests, Glass’s Delta effect sizes, and Spearman rank correlation.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Diffusion Models for Vision

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Operator learning has emerged as a promising tool for accelerating the solution of partial differential equations (PDEs). The Deep Operator Networks (DeepONets) represent a pioneering framework in this area: the "vanilla" DeepONet is valued for its simplicity and efficiency, while the modified DeepONet achieves higher accuracy at the cost of increased training time. In this work, we propose a series of Transformer-inspired DeepONet variants that introduce bidirectional cross-conditioning between the branch and trunk networks in DeepONet. Query-point information is injected into the branch network and input-function information into the trunk network, enabling dynamic dependencies while preserving the simplicity and efficiency of the "vanilla" DeepONet in a non-intrusive manner. Experiments on four PDE benchmarks -- advection, diffusion-reaction, Burgers', and Korteweg-de Vries equations -- show that for each case, there exists a variant that matches or surpasses the accuracy of the modified DeepONet while offering improved training efficiency. Moreover, the best-performing variant for each equation aligns naturally with the equation's underlying characteristics, suggesting that the effectiveness of cross-conditioning depends on the characteristics of the equation and its underlying physics. To ensure robustness, we validate the effectiveness of our variants through a range of rigorous statistical analyses, among them the Wilcoxon Two One-Sided Test, Glass's Delta, and Spearman's rank correlation.

Problem

• Accelerating PDE solutions with efficient operator learning
• Improving DeepONet accuracy without sacrificing training efficiency
• Introducing cross-conditioning for dynamic dependencies in networks

Innovation

• Transformer-inspired bidirectional cross-conditioning networks
• Dynamic dependencies with preserved simplicity and efficiency
• Equation-specific variant selection based on physics characteristics