MMET: A Multi-Input and Multi-Scale Transformer for Efficient PDEs Solving

2025-05-24

arXiv.org

AI Summary

Existing machine learning methods for solving multi-input, multi-scale partial differential equations (PDEs) suffer from poor generalization and high computational cost. To address these limitations, this paper proposes a novel Transformer-based architecture. Our method introduces four key innovations: (1) a dual-sequence structure that decouples grid points from query points; (2) a gated conditional embedding (GCE) layer enabling flexible incorporation of variable-dimensional physical inputs; (3) Hilbert-curve-based spatial reordering combined with block-wise embedding to compress sequence length; and (4) a multi-scale attention mechanism to enhance cross-scale feature modeling. Evaluated on multi-physics benchmarks, our approach achieves superior accuracy and inference efficiency over state-of-the-art methods. It significantly reduces computational overhead for large-scale geometric models and enables real-time, high-resolution PDE solving.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Partial Differential Equations (PDEs) are fundamental for modeling physical systems, yet solving them in a generic and efficient manner using machine learning-based approaches remains challenging due to limited multi-input and multi-scale generalization capabilities, as well as high computational costs. This paper proposes the Multi-input and Multi-scale Efficient Transformer (MMET), a novel framework designed to address the above challenges. MMET decouples mesh and query points as two sequences and feeds them into the encoder and decoder, respectively, and uses a Gated Condition Embedding (GCE) layer to embed input variables or functions with varying dimensions, enabling effective solutions for multi-scale and multi-input problems. Additionally, a Hilbert curve-based reserialization and patch embedding mechanism decrease the input length. This significantly reduces the computational cost when dealing with large-scale geometric models. These innovations enable efficient representations and support multi-scale resolution queries for large-scale and multi-input PDE problems. Experimental evaluations on diverse benchmarks spanning different physical fields demonstrate that MMET outperforms SOTA methods in both accuracy and computational efficiency. This work highlights the potential of MMET as a robust and scalable solution for real-time PDE solving in engineering and physics-based applications, paving the way for future explorations into pre-trained large-scale models in specific domains. This work is open-sourced at https://github.com/YichenLuo-0/MMET.

Problem

• Solving multi-input PDEs efficiently with machine learning
• Addressing multi-scale generalization challenges in PDE solutions
• Reducing computational costs for large-scale geometric models

Innovation

• Multi-input multi-scale transformer for PDE solving
• Gated Condition Embedding for variable dimension inputs
• Hilbert curve patch embedding reduces computational cost