LLT: Local Linear Transformer for PDE Operator Learning

2026-07-04

AI Summary

This work addresses the high computational complexity of standard Transformers and their difficulty in capturing local physical interactions when learning partial differential equation (PDE) operators. To overcome these limitations, the authors propose the Local Linear Transformer (LLT), which integrates linear global attention with a local spatial mixing mechanism and incorporates coordinate and geometric information to balance computational efficiency with physical locality. The method is applicable to both structured and unstructured grids. Evaluated on tasks including elasticity, plasticity, airfoil flow, pipe flow, and Darcy flow, LLT achieves comparable or lower relative L² errors than existing approaches, trains 1.8–2.5 times faster than Transolver, and successfully scales to a three-dimensional unstructured-grid aerodynamics dataset of an automobile with 32,186 nodes.

Abstract

Neural operators have become a common approach for learning PDE solution maps and accelerating numerical simulations. Transformer-based neural operators are of particular interest, since attention can learn long-range dependencies in the computational domain. However, standard attention has two major limitations when applied to PDEs: it scales quadratically with the number of computational nodes, and it lacks an explicit bias toward local interactions. To address these issues, we introduce Local Linear Transformer (LLT) for PDE operator learning. The architecture combines linear global attention with local spatial mixing, and incorporates coordinate and geometry information. We evaluate LLT on several PDE problems, including elasticity, plasticity, airfoil flow, pipe flow, and Darcy flow. The reference data for these problems span finite-element, finite-volume, and finite-difference discretizations on structured and unstructured meshes. Compared with other neural-operator and transformer baselines from prior studies, LLT achieves competitive or lower relative $L_2$ error across these problems. On matched structured discretizations, wall-clock time per training iteration is reduced by factors of 1.8 to 2.5 relative to Transolver. We also scale the approach and apply it to a three-dimensional car aerodynamics dataset with 32,186 unstructured mesh points per sample. Together, these results indicate that LLT provides an accurate and computationally efficient operator for PDE problems across discretizations, mesh types, and problem settings.

Problem

• PDE operator learning
• Transformer attention
• computational complexity
• local interactions
• neural operators

Innovation

• Local Linear Transformer
• PDE operator learning
• linear attention
• local spatial mixing
• mesh-agnostic modeling